Closed mouse-reeve closed 5 months ago
I have a tingly feeling this is the bug to discuss my failed export on bookwyrm.social.
Yes, almost certainly
I'm trying to understand what's happening here, and how difficult it will be to fix. A few questions:
A cleanup process for old exports was on the original list but we left it for another day. Possibly that needs to be revisted sooner rather than later.
Some things I've found that may or may not be helpful for this:
If I am remembering correctly, @dannymate did most of the work setting up Celery jobs, and @CSDUMMI did most of the work setting up the tarfile code, so they may be able to provide some suggestions.
It does unfortunately seem like a sooner rather than later issue. I'm seeing the disk usage creep up and then revert back to previous levels once the task fails -- I believe it's on disk not in RAM, but anything sys admin is in my weaker area of knowledge. The file does seem to remain on disk if the task runs successfully; I think this because I see gradual increases in disk usage in the monitoring that are consistent with and export as opposed to a backup, which would create a single jump in usage.
At the moment, the archive files are stored in FileField
s. Which appear to be stored on the local disk.
https://github.com/bookwyrm-social/bookwyrm/blob/a4599d0374e2719b50d7d381ca61fbb8a6a3f4cb/bookwyrm/models/bookwyrm_export_job.py#L26C19-L26C28
I was wondering about FileField
vs ImageField
, which is what everything else is stored as. Also I notice there's no upload_to
value set, though the Django docs are a bit unclear whether that's relevant.
I'll take a look at this tomorrow.
There's a couple logging lines in both bookwyrm_export_job and Bookwyrm_import_job. Not done by me.
If you'd like to go further. It's been a little while but, in terms of error handling and logging, you can override the on_failure method on Celery classes/tasks. If you want a generic handling of failure say to just log the exception then you can put this in job.py > Job class. You can also look at this example.
If you want more detailed logging say to log the size of an export you can put that closer to the logic say in BookwyrmExportJob.
If you want you can message me on Matrix or here and I can give it a shot if there's any information you specfically want logged.
Ok I've run a test in dev with Digital Ocean S3, and read @mouse-reeve's traceback.
If I'm reading the traceback correctly, this is an error with the database connection: it's saying it couldn't even properly mark the job as failed, because that means updating the job
instance and therefore connecting to the DB, but the connection had already closed (unexpectedly). Mouse is suggesting that this connection error seems to have happened because the host ran out of space.
I've run an (admittedly very small) use export in a dev environment and can confirm:
That leaves open the question of how much disk space is required whilst the export is running. I feel like that may not be the whole story though, because the traceback tells us that this is failing on json_export
:
File "/app/bookwyrm/models/bookwyrm_export_job.py", line 46, in start_export_task
json_data = json_export(job.user)
File "/app/bookwyrm/models/bookwyrm_export_job.py", line 121, in json_export
for edition in editions:
The json_export
function doesn't do anything with files - that's tar_export
, which runs after json_export
. The JSON export is just a text file and whilst sure, it might get big, it's unlikely to be a "large file" by any reasonable definition. If we take the traceback at face value, there's a problem with the for edition in editions
loop. Reading back through it, I can see we're making seven database calls within each iteration of the loop, which ...seems like a lot. So maybe this could be assisted/resolved by adding more select_related
collections in get_books_for_user
? i.e. could this be a problem of too many database connections within a single thread?
Not sure how exactly is it implemented in Bookwyrm, but connection also could be lost because of a simple timeout on DBMS side if something takes a very long time and makes no queries or keepalives in the mean time.
Resolved in #3228
Yay :tada:
Waiting for new release!
When a user export is running, it creates a very large file that is stored locally (rather than S3, when S3 is configured). This may be major blocker to the feature working, as it can hit the disk limit and cancel the task.