[RFC] Scene import process

[WithoutPants]

Problem

Need to provide a native data import process which handles mapping to other options. The process should allow for the creation of complete scene data without needing to perform post-import modifications.

Proposal

This RFC is based on the previous work in the branch at https://github.com/InfiniteTF/stash-box/tree/bulk-import

Here is the basic requirements I think are needed for the mvp import functionality:

• accept csv (json to come later)
• support mapping of csv columns to scene fields
• support fixed scene fields (not just studio)
• support manipulation of csv data via regex replacements
• support mapping studio/performer/tag names to existing objects

Here is how I envision the import process to work:

• user navigates to the import page
• user selects a file to import, configures scene field to column mappings (and fixed values), and clicks submit
• system parses the file, inserting the data into a database table, and attempts to match studio/tag/performer names to objects
• system presents the results and allows the user to map/change studio/tag/performer names from the data to actual object instances (using auto-complete selectors)
• user maps studio/tag/performer names to objects
• user clicks to submit
• system imports the data and removes the data from the import data table

Other considerations

• only one import operation may be performed per user. User may resume or abort processing of pending imports.
• stale pending import data should be removed eventually

Implementation details

Database schema

Add new import_data table:

• user_id - assuming a single import per user, this plus row should be sufficient composite key
• row - index of csv or json
• data - a jsonb encoded of the raw row data

If we need to identify stale imports, then we'll need a separate table to identify pending imports, with a user and date.

Graphql schema

• add submitImport mutation
  • accepts data file upload, a datafile type, and a list of fieldMapping objects
  • ImportFieldMappingInput includes outputField, optional inputField, optional fixedValue, optional list of regex replacements
  • processes the input file inserts into import table
• add queryImportData query, accepting page size and page. Returns parsed import data (does not translate tag/studio/performers).
• add importMappings query.
  • returns ImportMapping object, which contains studios, performers, tags fields, which are lists of name/id pairs
• completeImport mutation
  • accepts ImportMappingInput object which is basically the same structure as ImportMapping
  • creates scenes from the imported data, mapping tags/performers/studios as needed
  • names without a mapped id are ignored
• abortImport mutation
  • aborts a pending import for the current user, removing the data from the import data table

GUI changes

• add Import top-level menu item for allowed users
• if there is no pending import, then the import page will show the new import page, allowing selecting of data file and mapping of scene fields to data file fields
• clicking submit on this page submits the import and redirects to the pending import page
• pending import page is shown if there is a pending import for the current user
• pending import page allows browsing of the pending import data, and mapping of studio/performer/tag names to objects

Extensions for future iterations

• support manual data correction prior to importing
• support json files, mapping using gjson
• separate import role permissions and allow submitting imports with a separate user approving and importing the data

[bnkai]

A few thoughts i had concerning the import CSV functionality. (Didnt mention in the PR since they might be out of scope)

• When importing a csv file a check of the file should be performed ( either by file size or by row count ). Plain users shouldnt be able to overload the server resources
• Images could be allowed as a separate file (eg in a zip file) for cases where the user wants to submit images not currently available or behind a paywall or with expiration links. The image field in that case could be a filename instead of a URL ?
• A validation of some fields might be handy after the user matches the fields with the headers (eg dates, urls) and the post processing/regex phase is completed
• Another option for fields with multiple elements(performers,tags) would be to accept json arrays. The current implementation works through that with user defined regexes (eg the tags example) we could use something like strings.HasPrefix to check for the [ and attempt to json parse the record value automatically. CSV files like that are what Webscraper produces (i posted some actual files in the stash-box discord channel around January). Scrappy btw (haven't actually used it) seems to use the delimiter approach
• Users should be able to submit stash exported data. That would be possible either by adding an extra export mode ( maybe a CSV file for the text meta and a zip file for the images) or by converting to CSV the current exported zip file in stash-box. A standalone utility converter is also possible

[laurus-lx]

Is there any mechanisms in place to handle deduplication?

If not - I would suggest using canonical URL for duplicate detection for scenes pages available on the web from the original publisher, and for older content - use set/scene/movie links on the indexing sites (this sites could be also used to strengthen duplicate rejection when publishers are recycling content).