This pull request is addressing a couple of issues related to #1064.
1064 has always been very strange because it shows as the database not getting updated. This is very confusing as it's essentially a silent failure. This pull request fixes a couple of issues that might explain why it is a silent failure.
It turns out if a scraper does not generate a data.sqlite file at all (or one doesn't get transferred from the docker container) then it's as if no change is made to the current database that's stored on the morph server itself. This is very weird and counter-intuitive.
So, this changes that so that if a scraper does not generate a data.sqlite file then the scraper fails with an error message.
It also addresses another problem where if there is a docker server error during the transfer of the sqlite file from the docker container to the server, it was getting treated as if there was no file. Again, not good.
So, basically, it's by no means certain that these changes are going to fix the problem or even get us closer to it but the problems are serious enough (and confusing enough) in their own right that they're worth fixing and maybe we'll get lucky and it will help with the main issue #1064.
This pull request is addressing a couple of issues related to #1064.
1064 has always been very strange because it shows as the database not getting updated. This is very confusing as it's essentially a silent failure. This pull request fixes a couple of issues that might explain why it is a silent failure.
It turns out if a scraper does not generate a
data.sqlitefile at all (or one doesn't get transferred from the docker container) then it's as if no change is made to the current database that's stored on the morph server itself. This is very weird and counter-intuitive.So, this changes that so that if a scraper does not generate a
data.sqlitefile then the scraper fails with an error message.It also addresses another problem where if there is a docker server error during the transfer of the sqlite file from the docker container to the server, it was getting treated as if there was no file. Again, not good.
So, basically, it's by no means certain that these changes are going to fix the problem or even get us closer to it but the problems are serious enough (and confusing enough) in their own right that they're worth fixing and maybe we'll get lucky and it will help with the main issue #1064.