Failed Batch Ingest of Student Event films

[egilchrist]

Ingest of 101 .mov files failed. Divided into smaller batch of 5 .mov films. Received email confirmation that "Your metadata package was validated and no errors were found. Your batch is now processing." However, films would not load. On Avalon, error message of ERROR: Amazon Elastic Transcoder could not transcode the input file because the format is not supported." for all 5 films."

[egilchrist]

Here's the link to the files in Avalon https://media.northwestern.edu/?f%5Bcollection_ssim%5D%5B%5D=Student+Event+Films

[egilchrist]

This is the spreadsheet with the smaller batch of 5 films.Batch_manifest_student_events_batch1.xlsx

[egilchrist]

All of the films are in lala in dms28 in various folders...

[davidschober]

@mbklein it looks like this is an unsupported file format. @egilchrist where did these files come from?

[carrickr]

@mbklein, I'm on community sprint, but maybe Apple ProRes, which is currently not supported.

https://aws.amazon.com/elastictranscoder/faqs/

[jenyoung]

.mov files should be accepted I thought. Or is it how they were generated?

[egilchrist]

@davidschober - these were the access master files generated by Josh when he digitized the Student Event films from Archives. I don't know too much more about how they were digitized.

[davidschober]

Can you point us to where they live? @carrickr thanks for jumping in!

[egilchrist]

Hi. The files are in lala - in dms28. They are in the subfolder "Film_Deliverables" in the folder "NU-Student-Events-Media"

[egilchrist]

@davidschober Here's the progress on the the test Student Event films.

I ingested the two test files made by Josh into Avalon.

1) I received the email that the Batch was successful Your metadata package was validated and no errors were found. Your batch is now processing. Manifest file: s3://avalon-masterfiles-cfx3wj9/dropbox/Student_Event_Films/test/Batch_manifest_student_events_test_formats.xlsx

2) The .mov file caused an error again: “ERROR: Amazon Elastic Transcoder could not transcode the input file because the format is not supported.”

https://media.northwestern.edu/media_objects/m900nv04s

3) The .mkv file was successfully ingested.

https://media.northwestern.edu/media_objects/xs55mc74w

This is the email I got about the failure and success of the ingest

[davidschober]

@egilchrist and @jenyoung this seems to be related to apple pro res. Josh said he would retranscode.

[davidschober]

For Documentation:

https://aws.amazon.com/elastictranscoder/faqs/

Q: Do you support Apple ProRes or digital cinematography formats?

We do not support reading Apple ProRes files or raw camera formats like ARRI and RED at this time.

@jyocum can you put these into a format that we can ingest?

@egilchrist until we update batch 100 is about the limit of an ingest, you might want to break it up into two or three batches.

[davidschober]

We're digging into how to get these there. We're closing this and putting up an issue about >100gig.

[egilchrist]

After uploading the access copies to CyberDuck, I attempted to batch ingest the 101 Student Event films. 63 films made it into Avalon and are viewable.

38 files did not ingest.

I received this error message via email: Initial processing of your batch ingest failed. Details are below. Please see the Batch Ingest Package Format for further help.

Manifest file: s3://avalon-masterfiles-cfx3wj9/dropbox/Student_Event_Films/assets/Batch_manifest_student_events_revised_ac_mp4.xlsx

Ldp::HttpError: STATUS: 500 org.modeshape.jcr.value.binary.BinaryStoreException: org.modeshape.jcr.value.binary.BinaryStoreException: Unable to find binary value with key "60687b28ec98298b5dc69d595b5525d56b324967" within binary store at "/var/cache/tomcat8/temp/modeshape-binary-store" at org.modeshape.jcr.value.binary.S3BinaryStore.storeValue(S3BinaryStore.java:213) at org.modeshape.jcr.value.binary.AbstractBinaryStore.storeValue(AbstractBinaryStore.java:251) at org.modeshape.jcr.value.binary.BinaryStoreValueFactory.create(BinaryStoreValueFactory.java:257) at org.modeshape.jcr.value.binary.BinaryStoreValueFactory.create(BinaryStoreValueFactory.java:49) at org.modeshape.jcr.JcrValueFactory.createBinary(JcrValueFactory.java:149) at org.modeshape.jcr.JcrValueFactory.createBinary(JcrValueFactory.java:41) at org.fcrepo.kernel.modeshape.FedoraBinaryImpl.setContent(FedoraBinaryImpl.java:178) at org.fcrepo.http.api.ContentExposingResource.replaceResourceBinaryWithStream(ContentExposingResource.java:612) at org.f...

[egilchrist]

[egilchrist]

Batch_manifest_student_events_revised_ac_mp4.xlsx

[egilchrist]

Jen suggested that I try to re-ingest the films that failed. I just attempted it. Success! I received an email that "Your metadata package was validated and no errors were found. Your batch is now processing" and that the Student Events films were ingested. Thanks for all the help!