search

sul-dlss / preservation_catalog

Rails application to track, audit and replicate archival artifacts associated with SDR objects.
https://sul-dlss.github.io/preservation_catalog/
Other
2 stars 2 forks source link

preservation catalog worker boxes seem unable to write to zip temp space #1411

Closed honeybadger[bot] closed 4 years ago

honeybadger[bot] commented 4 years ago

Backtrace

[preservation_catalog/prod] Errno::EACCES: Permission denied @ dir_s_mkdir - /sdr-transfers/wr

line 44 of [PROJECT_ROOT]/app/models/druid_version_zip.rb: block in ensure_zip_directory!
line 44 of [PROJECT_ROOT]/app/models/druid_version_zip.rb: tap
line 44 of [PROJECT_ROOT]/app/models/druid_version_zip.rb: ensure_zip_directory!

View full backtrace and more info at honeybadger.io

also, a bit of manual testing:

[pres@preservation-catalog-prod-04 ~]$ touch /sdr-transfers/jmartin-test-file
touch: cannot touch `/sdr-transfers/jmartin-test-file': Permission denied

i'd expect a different error message, but just to be sure, made sure space wasn't an issue...

[pres@preservation-catalog-prod-04 ~]$ df -h /sdr-transfers
Filesystem            Size  Used Avail Use% Mounted on
sf6-sdr:/sdr_transfers
                       40T   20T   21T  50% /sdr-transfers
[pres@preservation-catalog-prod-04 ~]$ 
[pres@preservation-catalog-prod-04 ~]$ df -i /sdr-transfers
Filesystem             Inodes IUsed    IFree IUse% Mounted on
sf6-sdr:/sdr_transfers
                     85004503   556 85003947    1% /sdr-transfers

also

[pres@preservation-catalog-prod-04 ~]$ 
[pres@preservation-catalog-prod-04 ~]$ ls -lah /
...snip...
drwxr-xr-x   71  503  503  32K Feb 28 15:54 sdr-transfers

...snip...
[pres@preservation-catalog-prod-04 ~]$

i believe this wedges replication till it's fixed.

jmartin-sul commented 4 years ago

two failed jobs so far, one picked up by -02, one by -04. retried one, got the same error (picked up again by -02).

jmartin-sul commented 4 years ago

result of a now-fixed puppet config mistake, closing...