search

yaqwsx / jlcparts

Better parametric search for components available for JLC PCB assembly
https://yaqwsx.github.io/jlcparts/
MIT License
524 stars 49 forks source link

sqlite3.OperationalError: database or disk is full #132

Open Bouni opened 3 weeks ago

Bouni commented 3 weeks ago

Hi,

I just realized that you encounter the exact same issue as I do with my KiCAD plugin as we use your cache.

https://github.com/yaqwsx/jlcparts/actions/runs/9771283483/job/26973813599#step:5:258

My guess is that the GH actions runners have disk that is now insufficient to store the database file ...

Do you have a plan on how to solve this?

I lack the time at the moment, but I would really like to build an API that serves JLC parts data, sanitized and usable.

dougy83 commented 2 weeks ago

If you combine the multipart zip files into one file, you can mount the zip file using fuse-zip, and then use the mounted cache.sqlite3 as normal. The speed is pretty good too. E.g.

cat cache.z* > cache-combined-bad.zip
echo -e 'z\r\n' | zip -FF cache-combined-bad.zip -O cache-combined.zip
mkdir /tmp/cachezip
fuse-zip cache-combined.zip /tmp/cachezip
jlcparts buildtables --jobs 0 --ignoreoldstock 30 /tmp/cachezip/cache.sqlite3 web/build/data
fusermount -u /tmp/cachezip
Bouni commented 2 weeks ago

I'll look into yor suggestion as soon as I have a bit of spare time 👍🏽

yaqwsx commented 6 days ago

As a quick work-around, I tried to save some space on the CI machine. At the moment, it has been working just fine for 4 days now so the problem is solved - at least for now. I like the proposal of @dougy83, and I will look into it in the future.

dougy83 commented 2 days ago

@yaqwsx I've put the changes in PR #136 . I did mention that I came across write issues/corruption when using fuse-zip, but haven't been able to reproduce, and writing seems to work cleanly (not sure what I stuffed up before).