daitss / core

DAITSS: Dark Archive In The Sunshine State
GNU General Public License v3.0
9 stars 2 forks source link

Refresh error durring the validate aip descriptor step #807

Open szanati opened 6 years ago

szanati commented 6 years ago

I received the following error while refreshing a batch of packages:

time 2018-09-25 13:44:59 -0400

could not expand tarball=/var/daitss/data/work/E1JC9A9LC_4EJQWG/xmlresolutions/xmlres-0.tar into dir= /var/daitss/tmp/d20180925-2919-11zrlhc: pid 3131 exit 2 /opt/web-services/sites/core/releases/20180510000001/lib/daitss/proc/wip/tarball.rb:50:in block (3 levels) in make_tarball' /opt/ruby-1.9.3-p545/lib/ruby/1.9.1/tmpdir.rb:83:inmktmpdir' /opt/web-services/sites/core/releases/20180510000001/lib/daitss/proc/wip/tarball.rb:47:in block (2 levels) in make_tarball' /opt/web-services/sites/core/releases/20180510000001/lib/daitss/proc/wip/tarball.rb:40:ineach' /opt/web-services/sites/core/releases/20180510000001/lib/daitss/proc/wip/tarball.rb:40:in block in make_tarball' /opt/web-services/sites/core/releases/20180510000001/lib/daitss/proc/wip/tarball.rb:16:inchdir' /opt/web-services/sites/core/releases/20180510000001/lib/daitss/proc/wip/tarball.rb:16:in make_tarball' /opt/web-services/sites/core/releases/20180510000001/lib/daitss/proc/wip/refresh.rb:24:inblock in refresh' /opt/web-services/sites/core/releases/20180510000001/lib/daitss/proc/wip/journal.rb:16:in step' /opt/web-services/sites/core/releases/20180510000001/lib/daitss/proc/wip/refresh.rb:24:inrefresh' /opt/web-services/sites/core/releases/20180510000001/lib/daitss/proc/wip/process.rb:82:in block in spawn' /opt/web-services/sites/core/releases/20180510000001/lib/daitss/proc/wip/process.rb:66:infork' /opt/web-services/sites/core/releases/20180510000001/lib/daitss/proc/wip/process.rb:66:in spawn' /opt/web-services/sites/core/current/bin/pulse:161:inblock in start_wips' /opt/web-services/sites/core/current/bin/pulse:158:in each' /opt/web-services/sites/core/current/bin/pulse:158:instart_wips' /opt/web-services/sites/core/current/bin/pulse:194:in block in ' /opt/web-services/sites/core/current/bin/pulse:192:inloop' /opt/web-services/sites/core/current/bin/pulse:192:in `

szanati commented 6 years ago

This issue is similar to https://github.com/daitss/core/issues/802. I reset the package, E1JC9A9LC_4EJQWG/AA00012381_00001, and received the same error. I aborted the package and resubmitted the refresh package and received the same error. I also received the same error for packages: E1MF9V84T_MCIMUU/AA00013476_00001 and E1IHQGEVD_FKAF2T/AA00013233_00001.

cchou commented 6 years ago

This is a different problem than issue #802 though it may appear so. In issue 802, the tar file was a bogus tar (0 byte) so I added a tar validation step before extracting the tar to skip invalid tar.

The problem this time is the tar validation return xmlres-0.tar as being valid, but when daitss try to extract the file, it says the tar 'contain obsolescent base-64 headers". It looks to be a valid tar and I can the tar extraction is creating a directory for it, but it returns an error code.

[cchou@fclnx30 ~]$ tar -tf xmlres-0.tar E1MF9V84T_MCIMUU/manifest.xml E1MF9V84T_MCIMUU/http/digital.uflib.ufl.edu/metadata/sobekcm/sobekcm.xsd E1MF9V84T_MCIMUU/http/www.fcla.edu/dls/md/daitss/daitss.xsd E1MF9V84T_MCIMUU/http/www.loc.gov/standards/mets/mets.xsd E1MF9V84T_MCIMUU/http/www.loc.gov/standards/mods/v3/mods-3-4.xsd E1MF9V84T_MCIMUU/http/www.loc.gov/standards/mods/xml.xsd E1MF9V84T_MCIMUU/http/www.loc.gov/standards/xlink/xlink.xsd E1MF9V84T_MCIMUU/http/www.w3.org/2001/xml.xsd E1MF9V84T_MCIMUU/http/www.w3.org/2001/XMLSchema.xsd E1MF9V84T_MCIMUU/http/www.ukoln.ac.uk/repositories/eprints-application-profile/epdcx/xsd/2006-11-16/ E1MF9V84T_MCIMUU/http/www.w3.org/2001/03/xml.xsd

[cchou@fclnx30 ~]$ tar -xf xmlres-0.tar tar: Archive contains obsolescent base-64 headers tar: Skipping to next header tar: Error exit delayed from previous errors

Searching on the internet seems to suggest it mostly likely caused by file transfer error or hardware failure. Can you curl the package tar directly and put it on daitss-test?

szanati commented 6 years ago

Carol,

I was able to curl the tar. I put both E1MF9V84T_MCIMUU.tar and E1JC9A9LC_4EJQWG.tar on daitss-test. I change the permissions so you should be able to down load them.

cchou commented 6 years ago

Given my contract is ending, James, can you take this issue over?

szanati commented 6 years ago

Thanks Carol for all your help. I let James know.

jhostickflvc commented 6 years ago

Yes, I will be taking over this issue