Improve background processing

[elia]

• Attachment processor: transfer the files in separate jobs
• Process rows in the background to improve concurrency
• Group the rows by common changes (ex. for products by product slug)

Originally part of the roadmap in #5

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It might be closed if no further activity occurs. Thank you for your contributions.

[cesartalves]

Process rows in the background to improve concurrency

This would be extremely useful given that, with long files, the background processing library (eg Sidekiq) might crash in the middle and leave unfinished rows. Since the status of the SolidusImporter::Import is left as processing, these never are processed. This has happened a few times on a client project 🥲

Not only that, the way we import Orders right now is super fragile - we import all of them in an after_import call, many times breaking without a visible error, since there's only a context[:success] = false there which provides no useful error messages whatsoever

For files with thousands of lines, we're also building a huge context hash since we need it all in memory for the after_import call.

Since we already have the rows as raw Hash data, I don't understand why we need all the rows to have the context of the import from the beginning. We only need the context from related rows.

I sketched a still early draft solution of a possible solution for both problems: https://gist.github.com/cesartalves/df919b261012499ed30dfd11a6a1a672

• Move processing of related Rows to a Job
• Move the after_import call of the importer to such a job (would probably need to rename this slightly, maybe to after_group_import, but since the Order importer is the only one that actually implements this, I don't see why not!)

I may write a Draft PR soon!