search

julia-vscode / SymbolServer.jl

Other
23 stars 31 forks source link

Use JSON for the store #133

Closed davidanthoff closed 3 years ago

davidanthoff commented 4 years ago

Not a serious attempt, mostly I just wanted to see what is stored in these files ;)

But we might want to use something like this down the road.

davidanthoff commented 4 years ago

@ZacLN I think this now saves things properly, but not 100% sure. Maybe you could try it and take a look at the JSON files it generates? I'm not positive that they include all the info that is in the serialized files currently.

Right now it seems to me that the JSON version of any given cache file is about 4-5 times larger than the serialized version. When I index all of Queryverse.jl I have 114 packages in total, and that takes about 40 MB on disc, which seems very reasonable to me. These files seem to compress very well: the largest JSON file on disc takes 14 MB, when I zip compress it it takes 440 KB.

So my sense right now is that if we were to use JSON as the format for the cache, and computed those cached files in the cloud and then stored them in some compressed format in the cloud, downloaded the compressed version and decompress on the client, we would have a very reasonable solution in terms of file sizes and download speeds.

ZacLN commented 4 years ago

Yep I'll have a look - I was talking nonsense the other day in relation to the size of the caches (I was thinking of files sizes prior to https://github.com/julia-vscode/SymbolServer.jl/pull/128).

ZacLN commented 4 years ago

It looks to be storing everything (bar the TypeOfBottom issue). Unpacking the JSON may be a little bit involved?