search

uvalib / tracksys

UVA Library Digitization Services workflow and repository management tool.
8 stars 2 forks source link

Problems with /digiserv-migration finalization ( looks like an .AppleDouble problem ) #183

Closed sdm7g closed 6 years ago

sdm7g commented 10 years ago

We're getting failures on trying to finalize digiserv-migration. issue #182 fixes cleared up most of the .AppleDouble issues, but I believe this is related. Email of my theory pasted in below. We may be able to bypass the problem my changing the move in tracksys into a copy, followed by a move to ready_to_delete folder on the same filesystem. ( Maybe we'll still get errors there, but at least finalization will be successful. )

====== email to plp2j: ======

OK. I’ve cleared up some of my confusion.

The migration finalization copies the files into /digiserv-production/finalization/20_in_process . ( I had thought that the stuff on /digiserv-migration/finalization stayed there through it’s processing. )

I’m guessing the incomplete file count in the drop off directory is due to the fact that it’s getting an error during that move because the contents of the .AppleDouble directories are owned by ‘root’ . It can copy all of the files, but on move to a different filesystem, it is unable to delete those files from drop off. It finishes the copy, but dies in the middle of the delete, so all of the files are (probably) in 20_in_process, but some have been deleted from 10_dropoff , thus the incomplete count.

When I try to attempt the finalization a second time, it again tries to copy the directory, but since there is already an existing one, it copies that into the other so that there is a nested copy 000029785/000029785/ ( This is a weird feature of the unix ‘mv’ command due to it’s trying to allow both move’s and rename’s using the same command. )

If I had not tried it a second time, in theory, you could have proceeded to the QA_Unit_data button. ( This normally get’s launched at the end of the first finalization step, but since it’s dies on that ‘mv’ attempt, it never gets to launch the next step. )

I removed that duplicate directory and did the QA_Unit_data manually.

It was getting errors due to unknown files : there was a directory named MatchColor_000029785_Errors

Do you know where that came from ? ( Are you using a newer version of CaptureOne ? or some other software that is producing those files ? )

I removed that file and tried it again. It’s getting further but with these errors:

Success 2014-09-18 18:37:49 -0400 - Unit 29785 has passed the QaUnitDataProcessor. Failure 2014-09-18 18:38:01 -0400 - The of 000029785_1156.tif is an endpaper but does not specify 'front' or 'rear' in conformance with metadata standards: Paste-down Endpaper Error 2014-09-18 18:38:01 -0400 - Unit 29785 has failed the Filesystem and Iview XML QA Processor The of 000029785_1156.tif is an endpaper but does not specify 'front' or 'rear' in conformance with metadata standards: Paste-down Endpaper

Also: I don’t see an iView .xml file, just the .ivc catalog. I believe it requires one for the next step.

( Or was there an .xml file that got lost somehow in that copy/move problem. I didn’t see one in either directory, so I don’t think that was the case. )

I haven’t yet looked at how to fix these problems, but at least I now have some idea of what and how it’s going wrong.

Will look at it further tomorrow afternoon.

— Steve.

louffoster commented 6 years ago

digiserv-migration has been repurposed to support the new medium rare workflow. It has been in use for a few months now, and no issues like this have cropped up. Closing