facebookresearch / HolisticTraceAnalysis

A library to analyze PyTorch traces.
http://hta.readthedocs.io
MIT License
306 stars 43 forks source link

Fix Ijson metadata reader corner cases #165

Closed briancoutinho closed 4 months ago

briancoutinho commented 4 months ago

What does this PR do?

The fast ijson reader has some bugs/corner cases fixed here

  1. Device info = [] , this can happen on CPU trace
  2. nested arrays and maps inside the 2nd level metadata like "distributed Info". For now we are going to ignore this and support them in detail later.
  3. Fix param import test

Testing: Added a cpu only trace and unittest for these corner cases. pytest tests/test_trace_parse.py -k test_ijson_metadata_reader

Before submitting

facebook-github-bot commented 4 months ago

@briancoutinho has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

facebook-github-bot commented 4 months ago

@briancoutinho has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

fengxizhou commented 4 months ago

LGTM.

facebook-github-bot commented 4 months ago

@briancoutinho has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

facebook-github-bot commented 4 months ago

@briancoutinho merged this pull request in facebookresearch/HolisticTraceAnalysis@dcd33669d383f812e626ead8e2ebbe1a08f9bd73.