performance of writer turned out to be extremely slow (writing <100MB file with validation + several minutes which was previously maybe 30s), observed that chunking is very finely granularized

FAIRmat-NFDI / pynxtools

https://fairmat-nfdi.github.io/pynxtools/

Apache License 2.0

12 stars 8 forks source link

performance of writer turned out to be extremely slow (writing <100MB file with validation + several minutes which was previously maybe 30s), observed that chunking is very finely granularized #276

Closed mkuehbach closed 1 month ago

mkuehbach commented 1 year ago

checking the execution flow turned out that 99% of the time in the NXS writer is spent in not doing I/O, so yes auto chunker choices are not optimal but this is not the culprit, let's check with Rubels improvements

mkuehbach commented 2 months ago

[x] Was found out to be substantially dependant on verification, fixes in several places have been implemented but more work would be beneficial to make parsing faster in the future
[ ] Chunking

mkuehbach commented 1 month ago

Refactoring of verification https://github.com/FAIRmat-NFDI/pynxtools/pull/323 fixed performance issues