scientist-softserv / britishlibrary

Other

3 stars 0 forks source link

S3 backed Fedora timing out on large import #392

Open kirkkwang opened 1 year ago

kirkkwang commented 1 year ago

Summary

Related to:

On large imports, the S3 backed Fedora is timing out.

Acceptance Criteria

[ ] TheBlackpoolGazetteHerald.csv should import all remote files

Notes

Braydon worked on a new fcrepo image that may solve this.

cziaarm commented 10 months ago

Hi @kirkkwang

would you be able to expand on the details of this one for me. I'm seeing if any of these unexpected results for bigger files (#465 #173) might be more connected than I thought.

Thanks,

Rory

kirkkwang commented 10 months ago

@cziaarm from what i remember, we saw that in production when importing, because it would take so long to import the big files, the jobs from Sidekiq would just clear out after some time without any visible errors. What we deduced was that it was a bug of the S3 backed fedora where it just silently fails after a certain amount of time elapsing. I'm afraid that's all I can recall about this one.

cziaarm commented 9 months ago

fcrepo upgrade potentially fixes this (needs tested)

cziaarm commented 2 months ago

see #534