BatchUpload fails/hangs with local files

[lljohnston]

I just have not been able to get BatchUpload to work with local files on the PC. It gets to Step 3 of 4 and just sits there, process indicator flashing, never showing me any files or uploading them. I suspect I’m doing something simple wrong, like not properly specifying the path to the files on the local computer. The manual is silent on how it expects the files to be identified in the CSV - what structure is expected?

[dicksonlaw583]

The files are identified using a CSV column that is matched to special property "File". Here is an example:

Title,Filename
File One,001.jpg
File Two,002.jpg

After selecting this CSV file, map "Title" to "Title" and "Filename" to "File". When you proceed, you will be asked to upload 001.jpg and 002.jpg.

As an added note, this plugin uses the long-running job queue, which may require special setup in certain environments. Please verify its functionality with Job Diagnostics.

[lljohnston]

That's what I had:

File,Item Type,Collection,Creator,Year,Con,Location,Group,Type,Pictured,Rights 1979MythconDawnCleric.jpg,Still Image,Leslie Johnston,Dawn Davidson,1979,Mythcon,"San Jose, California",Cleric,Hall Costume,Dawn Davidson,Leslie Johnston 1979MythconLeslieElf1.jpg,Still Image,Leslie Johnston,Leslie Johnston,1979,Mythcon,"San Jose, California",Elf,Hall Costume,Leslie Johnston,Leslie Johnston

Where "File" is mapped to File for the upload and the others are mapped to appropriate DC fields including Title.

Running JobDiagnostics I get no error messages related to the specific jobs. I do get this:

"Latest result (Nov 8, 2018): Response in 18000 seconds. Queue nearing capacity, or re-test required."

Several jobs I began never finished and appear to be still running, including one that's been running for 2 weeks. None show up in the Jobs tab under Batch Uploader to delete them. How do I stop the job processes when I can't see them through the interface?

[dicksonlaw583]

Try the long-running job test now and see if you get a response within 10 seconds. Sometimes the job dispatcher could become too busy and not get a chance to execute new jobs. If a job somehow disappears from Batch Uploader (hasn't occurred before in testing before), you can always try again with the same CSV file.

You can use the Processes tab in the Job Diagnostics plugin to see which jobs are taking longer than usual. If you have a lot of content, certain jobs like reindexing can take a long time, but probably not on the scale of weeks.

Also, if logging is enabled on your installation, can you look in it and see if there are any errors mentioning the Batch Uploader or other plugins? You can find it in applications/logs/errors.log under the Omeka installation directory.

[lljohnston]

I'm circling back to this after some time away. I used the CSV uploader to solve an immediate load need, but going back to this I see the same issue - it never gets to step 4 where it confirms the files. This is for a load with 11 files in it.

Nothing's written to my log yet, but the Job Diagnostics say it's still running and I see this message: Latest result (Feb 21, 2019): Response in 18000 seconds. Queue nearing capacity, or re-test required.

I'm running this in a hosted environment - Reclaim - and I'm not sure what to do at this point. I'm doing this on a volunteer basis for a non-profit, and haven't done any hard-core development or sys admin in years.

[lljohnston]

It definitely finds and reads the csv, and the files are in the same directory and are identified without paths.

[lljohnston]

OK, I just checked my error log:

[21-Feb-2019 17:26:26 UTC] PHP Fatal error: Uncaught Error: Class 'BatchUploadJob' not found in /home/gallery1/public_html/application/libraries/Omeka/Db/Table.php:682 Stack trace:

0 /home/gallery1/public_html/application/libraries/Omeka/Db/Table.php(670): Omeka_Db_Table->recordFromData(Array)

1 /home/gallery1/public_html/application/libraries/Omeka/Db/Table.php(214): Omeka_Db_Table->fetchObject(Object(Omeka_Db_Select), Array)

2 /home/gallery1/public_html/application/libraries/globals.php(1878): Omeka_Db_Table->find(11)

3 /home/gallery1/public_html/plugins/BatchUpload/libraries/BatchUpload/Job/GenerateRows.php(49): get_record_by_id('BatchUpload_Job', 11)

4 /home/gallery1/public_html/application/libraries/Omeka/Job/Process/Wrapper.php(29): BatchUpload_Job_GenerateRows->perform()

5 /home/gallery1/public_html/application/scripts/background.php(61): Omeka_Job_Process_Wrapper->run(Array)

6 {main}

thrown in /home/gallery1/public_html/application/libraries/Omeka/Db/Table.php on line 682 [21-Feb-2019 17:38:09 UTC] PHP Warning: Declaration of BatchUpload_MappingSetsController::_redirectAfterAdd() should be compatible with Omeka_Controller_AbstractActionController::_redirectAfterAdd($record) in /home/gallery1/public_html/plugins/BatchUpload/controllers/MappingSetsController.php on line 8 [21-Feb-2019 17:38:09 UTC] PHP Warning: Declaration of BatchUpload_MappingSetsController::_redirectAfterEdit() should be compatible with Omeka_Controller_AbstractActionController::_redirectAfterEdit($record) in /home/gallery1/public_html/plugins/BatchUpload/controllers/MappingSetsController.php on line 8

[dicksonlaw583]

The CSV reading part takes place on the client side, so that should work.

Can you enable debug logging (details here) on your installation and retry with a new upload job? I would need to know more about why your installation is looking for a BatchUploadJob model (it should actually be BatchUpload_Job and has been that way in my test environment) and what is causing it to jump to a position that indicates an error with the job (line 49 in GenerateRows.php is in a catch statement).

Also, what is your PHP version? If it is 5.5 or below, please try with 5.6 or above as that is what we used to test. See this forum post for details.

[lljohnston]

My PHP version is 7.0.33. I enabled debugging logging but it's not writing to the log, even though I have my permissions set to 774 on the public_html/application/logs/errors.log file

This has close to reached my ability to troubleshoot. I haven't done any dev work in a very long time - I'm doing this on a volunteer basis for a non-profit.

[dicksonlaw583]

I have taken your sample from here and got an error on my development environment about collection names. For the "Collection" value type, the current setup requires an exact name match. I can try patching it to loosen up the requirement and make it not halt the import. That would leave them unattached and allow you to manually attach them to a collection via a batch edit later.

As a further diagnostic measure, please provide additional samples of failed import CSV files (no need for the actual JPG files, as the goal is to reach the image upload screen). I will try to work on this issue over the weekend. My apologies for the inconvenience that the plugin has caused.

[lljohnston]

File,Item Type,Public,Featured,Contributor,Original Format,Collection,Title,Creator,Date,Description,Coverage,Spatial Coverage,Type,Rights Holder,Tags 201110501-2937.jpg,Still Image,Yes,Yes,Leslie Johnston,Costume,Leslie Johnston Collection,Lady Godiva Ribbon Weave Vest,Jeannette Holloman,2011,Jeannette Holloman,CostumeCon,"Hasbrouck Heights, New Jersey",Single Pattern Competition,Unknown,"CostumeCon,Single Pattern Competition" DSC_7347.jpg,Still Image,Yes,Yes,Leslie Johnston,Costume,Leslie Johnston Collection,FutureIkons,Jeannette Holloman,2004,Jeannette Holloman,Worldcon,"Boston, Massachussets",F/SF Competition,Charles Mohapel ,"Worldcon,F/SF Competition" DSC_7348.jpg,Still Image,Yes,Yes,Leslie Johnston,Costume,Leslie Johnston Collection,FutureIkons,Jeannette Holloman,2004,Jeannette Holloman,Worldcon,"Boston, Massachussets",F/SF Competition,Charles Mohapel ,"Worldcon,F/SF Competition" ff20a.jpg,Still Image,Yes,Yes,Leslie Johnston,Costume,Leslie Johnston Collection,Peacock Format,Jeannette Holloman,1985,Jeannette Holloman,CostumeCon,"Columbia, Maryland",Future Fashion Folio,Unknown,"CostumeCon,Future Fashion Folio" ff20b.jpg,Still Image,Yes,Yes,Leslie Johnston,Costume,Leslie Johnston Collection,Peacock Format,Jeannette Holloman,1985,Jeannette Holloman,CostumeCon,"Columbia, Maryland",Future Fashion Folio,Unknown,"CostumeCon,Future Fashion Folio" ff22.jpg,Still Image,Yes,Yes,Leslie Johnston,Costume,Leslie Johnston Collection,Ceremonial Robes,Jeannette Holloman,2000,Jeannette Holloman,CostumeCon,"Hartford, Connecticut",Future Fashion Folio,Unknown,"CostumeCon,Future Fashion Folio" h02.jpg,Still Image,Yes,Yes,Leslie Johnston,Costume,Leslie Johnston Collection,The Sacking of Rome,"Elaine Mami, Diane Kovalcin, Karen Kovalcin, Steve Swope, Janice Dallas, Erma Johnson, Susan Toker, Susan Conner, Sharon Trembley (Kneeling, left to right): Gloria White, Signe Merrifield, Shanti Fader, Toni Lay, Jeannette Holloman, Denice Girardeau, Jeannie Johnson, and Madeline Dougherty",1997,"Elaine Mami, Diane Kovalcin, Karen Kovalcin, Steve Swope, Janice Dallas, Erma Johnson, Susan Toker, Susan Conner, Sharon Trembley, Gloria White, Signe Merrifield, Shanti Fader, Toni Lay,Jeannette Holloman, Denice Girardeau, Jeannie Johnson, Madeline Dougherty",CostumeCon,"Baltimore, Maryland",Historical Competition,Unknown,"CostumeCon,Historical Competition" h10.jpg,Still Image,Yes,Yes,Leslie Johnston,Costume,Leslie Johnston Collection,African Tudor,Jeannette Holloman and Jennifer Ketcham,1989,Jeannette Holloman,CostumeCon,"Albany, New York",Historical Competition,Unknown,"CostumeCon,Historical Competition" sf20.jpg,Still Image,Yes,Yes,Leslie Johnston,Costume,Leslie Johnston Collection,Undersea Circus Parade,"Deb Salisbury, Denice Giradeau, Jeannette Holloman, Suzanne Garcia, Ron Robinson & Carol Salemi",2000,"Deb Salisbury, Denice Girardeau, Ron Robinson, Jeannette Holloman, Suzanne Garcia, Pella Conner, Carol Salemi, Ann Lesnik, Sharon Landrum, Stephanie CarriggSigne Merrifield, Shanti Fader, Toni Lay, Jeannette Holloman, Denice Girardeau, Jeannie Johnson, Madeline Dougherty",CostumeCon,"Hartford, Connecticut",F/SF Competition,Unknown,"CostumeCon,F/SF Competition" sf30d.jpg,Still Image,Yes,Yes,Leslie Johnston,Costume,Leslie Johnston Collection,The Creation of Agalla,"Betsy Marks, Deborah Feaster, Sharon Landrum & Terelynn Marks",1993,Jeannette Holloman,CostumeCon,"Pittsburgh, Pennsylvania",F/SF Competition,John Upton,"CostumeCon,F/SF Competition" TZGroup.jpg,Still Image,Yes,Yes,Leslie Johnston,Costume,Leslie Johnston Collection,Fridays at Ten,"Amanda Allen, Michael Atkinson, Tom Atkinson, Dan Corcoran, Deborah Feaster, Jeannette Holloman, Sharon Landrum, Ric Rader, Ron Robinson, Don Sakers, Ann Stephens, and June Swords",2001,"Amanda Allen, Michael Atkinson, Tom Atkinson, Dan Corcoran, Deborah Feaster, Jeannette Holloman, Sharon Landrum, Ric Rader, Ron Robinson, Don Sakers, Ann Stephens, and June Swords",Worldcon,"Philadelphia, Pennsylvania",F/SF Competition,Ernest Lilley,"Worldcon,F/SF Competition"

[lljohnston]

It should be an exact value match for collection, so I'm looking to see how it might not match.

[lljohnston]

But hooray! While I separately try to figure out what didn't match, I was able to load 2 different ways: leaving Collection field in the CSV as Unmapped and loading the records as items or adding into an existing collection. Without the conflict it worked just fine. Thanks for helping me troubleshoot this!

[dicksonlaw583]

I have identified a potential fix for both the "Collection" field and "Class BatchUploadJob not found" problems. Could you please download this branch and let me know if you still get either errors?

https://github.com/utlib/BatchUpload/tree/hotfix-leslie-johnston

[dicksonlaw583]

Crash fixed in 1.1.0.