Suggestion: Dynamic lookup mode

[mortbopet]

It seems like VCDVCD struggles a bit with opening very large VCD files. My hunch is that this is due to the entire VCD trace is being loaded into internal datastructures upon loading the file, something which kills performance when loading >100MB files.

An alternative here could be to rework how the VCD file is being parsed:

1. Upon construction, index the module hierarchy and all of the signals
2. Upon lookup of a cycle value for a given signal:
   • binary search the VCD file (Read line at N trace lines / 2 and scan linearly backwards until hitting a timestep, and recurse down or upwards based on that timestep)
   • Each time a timestep is encountered, maintain a caching {timestep : file line} which can be used during lookup to quickly narrow in on a range of filelines where we expect the requested timestep to be.

Some additional caching could be added as well. The main goal would just be to amortize the cost of reading the VCD file until we actually need the values, so we don't have to wait multiple minutes (and use up all of our RAM!) with the VCD trace :+1: .

[cirosantilli]

Hi,

I see that this could be useful.

I just wonder how much more useful this would be than moving to a properly indexed file format, e.g. converting the VCD into a sqlite database with indexed rows (and I'm sure EDA vendors must have their own indexed formats, I wonder if there's a standard, but SQLite would likely work really well for this)

This would prevent blowing up RAM completely, and after conversion would make each lookup as fast as we can make them.

Yes, you do have to do a possibly lengthy conversion once. But I wonder how many single accesses you'd need to do until it would have been cheaper to just do a conversion first.

Pull requests for either approach will be considered.