Add a page to the docs about "Enhancing the Workbench Experience"

[mjordan]

Sorry about the cheezy title, we can do better than that. Suggestions welcome.

Intent of this documentation is to recommend some things that will lead to greater success using Workbench, or will help recover if something goes wrong. Some points to include are:

• Use an up to date version of Workbench
• Always use --check
• Use the perform_soft_checks: true config setting
• Use job-specific log file names using log_file_path config setting
• Break large jobs up into smaller jobs (e.g. instead of creating 5000 nodes at one, use 5x1000 node jobs); instead of adding all media to nodes during create tasks, create basic nodes+media and use a separate add_media task to add additional media.
• Using timestamp_rollback: true config setting
• Create a View that allows you to use the "Checking if nodes already exist" feature
• Ingesting subsets or samples from your input CSV using the "Using CSV row ranges" or "Processing specific CSV rows" features
• Consider techniques for speeding up Workbench as described at https://mjordan.github.io/islandora_workbench_docs/troubleshooting/#workbench-is-slow

Anyone else have items they want to suggest?

[khart20]

A user interface to workbench versus cli tools would be the ideal. But more realistically to help with the day to day... If there was a way to not have to edit the config file with input_csv output_csv names. Usually we're using the same .yml files and those fields are the two fields you have to edit. It would remove a step if part of the cli input to run the yml was setting those values.

[mjordan]

@khart20 nice feature. So something like python workbench --config foo.yml --input_csv myinput.csv --output_csv myoutput.csv?

[rosiel]

I enhance my workbench experience with

alias wb='workbench --config=config.yml'
alias wbc='workbench --config=config.yml --check'

Then I just organize my tasks into folders, and name the config files config.yml as a rule. That way I can just run wbc and wb instead of the long command.

[DonRichards]

Here is a suggestion (can you tell I ran it through a gpt for proofreading?)

1. Implement automatic retries when the source server returns a 500 error.
2. Introduce a progress tracking feature that allows users to resume migrations from the point of interruption.
3. Combine the create and update functions to streamline the process.
4. Automatically add missing fields or provide a migration option to update the "Islandora_object" content type.

Detailed Descriptions:

• # 1: Implement throttling and error tolerance mechanisms. Our current server frequently crashes under heavy load during migrations. The system should throttle requests and retry if the source server throws a 500 error.
• # 2: Introduce a feature to save migration progress. Currently, if the source server halts or times out, it disrupts the entire migration process, making it difficult to resume without starting over.
• # 3: Treat the addition of content to an object in a spreadsheet as a dynamic update, allowing the spreadsheet or CSV file to be updated at any time.
• # 4: Address the issue of missing fields in migrations—this is a complex issue, but essential for ensuring data integrity and completeness.

[DonRichards]

Enhance the current export functionality by allowing users to export all nodes without requiring a CSV of node IDs, and write directly to a Google spreadsheet.

This should include:

• Solr configurations and Islandora object fields: Exported to a separate sheet.
• Content: Collections and Islandora objects with all field content and media references.
• Media content: Details of media items.
• Files: All related information.

Each sheet should maintain references to related sheets/tabs, ensuring comprehensive data relationships. Users would provide the site URL and spreadsheet URL for export. This feature would streamline the export process and improve data management.