search

pulibrary / figgy

Valkyrie-based digital repository backend.
Other
36 stars 4 forks source link

Lots of FileAppender errors #6338

Closed tpendragon closed 5 months ago

tpendragon commented 5 months ago

There are hundreds of these starting in the middle of last week. The error gets ignored so nothing is retried. But this is still failing right now so before we can try ingesting them again we should figure that out and fix it.

[Figgy/production] Notice: FileAppender: Failed to append the new file 00000378.tif for 97d70fa8-dd46-4cbd-ba57-f1b4cf1fca78 to resource 4446b9b3-54a5-4d74-9e10-dd08c64fe1e7: Resource temporarily unavailable @ dir_s_mkdir - /opt/repository/files/97/d7/0f/97d70fa8dd464cbdba57f1b4cf1fca78

Backtrace

line 138 of [PROJECT_ROOT]/app/services/file_appender.rb: rescue in create_node
line 115 of [PROJECT_ROOT]/app/services/file_appender.rb: create_node
line 145 of [PROJECT_ROOT]/app/services/file_appender.rb: block in file_nodes

View full backtrace and more info at honeybadger.io

Sudden priority justification

Not sure how widespread but it seems like lots of ingests are not getting in.

tpendragon commented 5 months ago

I'm asking Operations if something changed on the 9th to cause this huge uptick, but maybe generally we need to put some file copy retry logic around https://github.com/pulibrary/figgy/blob/main/app/services/file_appender.rb#L124 and then get rid of error suppression?

hackartisan commented 5 months ago

Good idea.

tpendragon commented 5 months ago

We're pretty sure a kernel update caused this: https://forums.linuxmint.com/viewtopic.php?t=417371

Francis is working on pinning the Figgy boxes to an older version.

tpendragon commented 5 months ago

This is probably done, but we'll check the honeybadger issue tomorrow after ingests have been going to make sure it's fixed.