404 for openSNP data dump

[apriha]

When attempting to download the openSNP data dump, occasionally a 404 is returned.

Perhaps this could be solved by atomically writing the file in zipfulldata.rb, e.g., copy to a temp file at public/data/zip/ and rename it to the final file?

[gedankenstuecke]

I think the way it's currently tried to be resolved is by having a symbolic link that points to the correct file, but I guess sometimes it happens that the file isn't created but that the link is changed regardless of that. I'll try to find some time to look into it!

(Unrelated: @apriha could you check the OH slack, I had some questions re: Lineage there 😇 )

[nogilnick]

The link is 404ing again.

[philippbayer]

Hi! i just rebooted the machine running the dumping job - it looked like it had a hanging network connection to the shared filesystem, which is now back. the new dumping job is running and we might have a link again in a few hours

[nogilnick]

Unfortunately it seems the reboot did not fix the problem. Without logs, it's hard for me to do anything else than point out the painfully obvious :joy:.

[philippbayer]

thanks :) yes I found another issue - once the shared filesystem came back it now crashes looking for a single missing file when zipping up everything. I'm investigating what happened to that file, whether it was an incomplete database delete or something else. definitely very hard to debug without server access :)

[philippbayer]

Hi! The link works again. The data mount between sidekiq (zipping) machine and the web-server was lost, after restoring that link the Docker instance inside the sidekiq machine wasn't seeing it. So two levels of failure :) It's fixed now. Please let us know if it breaks again