CSV exports without lumen-csvexport-queues requires documentation or changes

[tuxpiper]

With the new release (master as of Apr 17 2018), we have deployed CSV exports to be handled through an additional component ( ushahidi/platform-lumen-csvexport-queues ). This method is not accessible for other deployers (open source community).

I'm proposing to either document how to obtain CSV exports without installation of the component or to study what changes need to be introduced to bring the functionality back.

Aha! Link: https://ushahiditeam.aha.io/features/PROD-365

[tuxpiper]

@willdoran do you have comments that could guide us on this one?

[rowasc]

@willdoran @tuxpiper maybe we could write documentation about how to use this with CLI exporter + a cronjob + a place to upload files? They would not need a queue for that. The main issue there is that they would need a way to join the files if they want to request in chunks, which would need something like what we do in lumen_csvexport_queues repo code (ConcatenationManager)

[rjmackay]

IMO I think the architecture for CSV exports should be refactored. The short version is:

• Remove the lumen_csvexport_queues entirely
• Remove csv export code from platform-job-queue
• Replace it with laravel's job queues
• Add some kind of trigger in the export job API that creates a Job (in the lumen queue)
• Migrate whats currently done in lumen_csvexport_queues into that Job ie.
  • Check count for CSV
  • Export each chunk
  • Merge chunks
  • Update status
• Most of that could happen directly, not via API requests now

This would basically mean deployers just have to run lumen's job queue

[willdoran]

The originally purpose, I think, of creating the export handler in Lumen was so that it could be merged back to platform when Lumen was in the API. So the above seems reasonable.

[rjmackay]

@jrtricafort we should consider implementing this in this cycle or the next one. It won't solve all the CSV issues but it might make them simpler to debug.

[jrtricafort]

Thanks for the flag Robbie. I'm hesitant to add this into the current cycle since it's already an aggressive scope. Can we do next cycle? @caharding @willdoran @Angamanga what do you think?

[willdoran]

@jrtricafort I would also suggest waiting.

[rowasc]

@jrtricafort Maybe cycle 7 would be a good place for this one?

[rowasc]

@kinstelli pointed out that this ticket is related to https://app.zenhub.com/workspace/o/ushahidi/platform/issues/2890 , so if we move it to cycle 7 they should probably go together.

[jrtricafort]

@rowasc Isn't this whole issue actually going to be resolved once Lumen is on Prod? If so, can we archive this? Let me know if I'm missing something here.

[rowasc]

no, this issue is unblocked by lumen , but doesn't get resolved by releasing lumen. Lumen gives us queues that we can use to do this work @jrtricafort

[rowasc]

@jrtricafort setting this up in cycle7 so we don't lose track , lets discuss in planning

[rowasc]

Romina to review current status of the PR and make a plan to get a OSS version of exports handler

[rjmackay]

@rowasc I'm claiming this to track the remainder of the multisite-job-queue work