yaqwsx
commented
7 months ago To clarify: there are four files per category. One holds the components and it is expected to update infrequently, the second one contains current stock. And each of them has a file with a hash to be able to check if it has changed or not.
The multiple files per category were introduced as the raw JSON files often had around 50-90 MB. Once we hit the GH-pages file-limit size, we implemented explicit compression (mainly to avoid hitting the limit of file size) and thus, probably made the whole mechanism contra-productive.
It's possible that we would even hit a smaller file size if we combined all the files into a single archive.
I am hesitant to make changes as I would like to put my energy into #37 which should resolve a ton of our problems. But feel free to open a PR!
PS: Note that the IndexedDB is also quite slow so it still might be worth it to not discard the whole DB but make partial updates as the components change infrequently.
dougy83
Updating the local component database makes >3000 requests, and takes a few minutes to download something like 20MB. An example download of a 1kB .json.gz resource took 242ms, with 238ms of that being waiting for the server to respond.
If all data was combined into a single file, the download should take only a few seconds.