As a content editor, I want to batch import new data on books' genre, so that I can update multiple records with annotations made externally

[quadrismegistus]

Notes:

• Try django-import-export for this one: is it sufficient?
• Requirements: log entries documenting change; log entry should match the importer; needs to handle conflicts
• Note: django-import-export may make this much easier since it tracks log entries natively and will flag conflicts (need to check this). Estimate may rise if this doesn't work

Update 2/1/24: We have this working now in the admin interface on QA using django-import-export. If you go to /admin/books/work you should see an import and export button. Export should export all Works, with all fields. Import should, for now, import only the fields relevant to the currently annotated data (genre category).

Testing instructions

• [x] Confirm import/export buttons are present on Work admin page
• [x] Confirm csv export works and that fields are present
  • [x] Confirm a 'slug' field is in export -- this field is needed for import
• [x] Import a csv of 10 new books' info
  • [x] Confirm a preview of changes is shown
  • [x] Click "confirm import" button works, check that number of records updated is shown
• [x] Import a csv of all new book info
  • [x] Check that preview of changes is shown
  • [x] Check "confirm import" works and shows number of records updated
  • [x] If more than 1,000 records updated, check that a warning message shows at top of page indicating that a server-side index will be necessary
  • [x] ~Confirm that new genre categories not in DB now are (Nonfiction)~ [This is working, but certain genres in annotation are multiple (Fiction; Poetry). I mistakenly thought we only wanted one per book. I will refactor if we want to preserve having multiple.]
  • [x] Confirm that books previously without genre category now have them when seen from the admin interface
  • [x] Confirm that log entries for updated Works show a change was made from django import export / admin interface
• [x] Check for appropriate errors
  • [x] Try re-importing the 10 book csv above but first editing it in excel to replace slug in the header with slugxx or another word; then make sure an error is shown
  • [x] Try re-importing a book CSV after all steps above made. Confirm that all records are skipped because already updated

[jkotin]

We have given a "genre" for each item/book in the lending library: drama, fiction, nonfiction, poetry, periodical, N/A. We should discuss/confirm that these are the best labels. We may also consider whether we should create a structure that would allow future work: e.g. dividing nonfiction into philosophy, history, etc. We don't have time to do that kind of dividing now.

[jkotin]

Small thing: there's a typo in "genre category" -- it's reads "Genre Cateogry."

[jkotin]

I received an error message when trying to do the larger import. Maybe I should do the import from campus? If so, I can try again on Tuesday.

[jkotin]

Two additional notes:

1. As you mention, we will want some books to have multiple "genre" categories.
2. Will we be able to edit the genre category on individual work pages in the database? This is important: in case we make a mistake and need to fix it. For example, we might mis-categorize a book as a novel when its true crime.

[jkotin]

I opened the test site this evening to finishing testing. The import went through despite the error message (see above) -- but the import added 23 "empty" works. See this screenshot.

[jkotin]

For this test:

Try re-importing the 10 book csv above but first editing it in excel to replace slug in the header with slugxx or another word; then make sure an error is shown

I did as instructed, but didn't see an error message.

[jkotin]

When I try to do this:

If more than 1,000 records updated, check that a warning message shows at top of page indicating that a server-side index will be necessary

I get this error message:

[jkotin]

Still receiving an error message.

[jkotin]

This is the CSV I used.

books-with-category(3).csv

[rlskoeser]

@jkotin I figured out what caused the error: your CSV had the heading 'category' but you and Ryan decided to set the database to allow multiple, so the import was expecting it to be called 'categories'. I've revised the input so it should work with either category or categories. Please confirm.

[jkotin]

I'm getting an error again, but I suspect that's because the VPN sometimes gets weird. I can try again from campus on Monday evening.

[jkotin]

I tried from campus and got the same error warning.

[rlskoeser]

@jkotin I see log messages indicates data is being imported and then it stops; no error, but I think it may be getting cut off by a timeout on the load balancer. I'll talk to PUL about increasing the timeout and let you know.

[rlskoeser]

@jkotin I've coordinated with PUL to increase the timeout on the load balancer, when you get the chance please test this again and report back on whether it works or if you're still getting the error.

[jkotin]

Yay! It worked.