cdyk / rvmparser

Code to work with AVEVA PDMS RVM files
MIT License
98 stars 55 forks source link

feature request - less memory #67

Closed VegarRingdalAibel closed 1 year ago

VegarRingdalAibel commented 1 year ago

Hi

Been having issues on very large files with current memory usage. Is it possible in combination with --output-gltf-split-level for it to just read rvm file per split lvl and clear used memory per lvl, so it stays low. Trying to use it in some cloud service, but container is getting kick due to memory usage.

vegarringdal commented 1 year ago

Made a small workaround for this issue

Sharing it here just in case anyone needs something similar. https://github.com/vegarringdal/rvmsplitter

This also extract title and date from the rvm file to a json fil, ref (#68)

Used Denojs, and made executable for windows and linux. Might make something in c++/rust later if I ever het around to learn it 😂

cdyk commented 1 year ago

I have looked through the code and run some tests, it doesn't look like there is a big memory leak, so the issue might simply be that loading a single large rvm file into memory takes too much memory?

Your approach with the splitter seems like a good solution!

VegarRingdalAibel commented 1 year ago

Yes it might be. Splitter have been nice so far. Might be a good place for me to play around with low level language. Since I know I should parse file better to make sure I dont skip part/tricked by text on names items in tree 😂

But have a lot of projects/rvm files to test it on.

Btw, thank you for making rvmparser and keeping it up to date with last changes you did. github actions was really hepful for making new build in my branch.

VegarRingdalAibel commented 1 year ago

Closing this, since its really not needed for my part.