Feature request: Automatically trigger export export when bulk import succeeds

[bmamlin]

For large imports (e.g., CIEL), it would be helpful if export generation could be automatically triggered after a bulk import succeeds. To avoid unnecessary export generation, perhaps this could be a "Generate export if import succeeds" opt-in feature for bulk imports

[paynejd]

If you add one more json line to the import file to create a temp version, it will automatically create an export. Would that meet the need, rather than implementing a feature?

On Thu, Jun 27, 2024 at 5:53 PM Burke Mamlin @.***> wrote:

For large imports (e.g., CIEL), it would be helpful if export generation could be automatically triggered after a bulk import succeeds. To avoid unnecessary export generation, perhaps this could be a "Generate export if import succeeds" opt-in feature for bulk imports image.png (view on web) https://github.com/OpenConceptLab/ocl_issues/assets/860834/ff849912-feeb-4702-83ad-0ab3a25ad1f6

— Reply to this email directly, view it on GitHub https://github.com/OpenConceptLab/ocl_issues/issues/1870, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAJCOOIPZXEYOH32F4NG6U3ZJSCXNAVCNFSM6AAAAABKAXL44OVHI2DSMVQWIX3LMV43ASLTON2WKOZSGM3TSMJUGYZDENQ . You are receiving this because you are subscribed to this thread.Message ID: @.***>

[bmamlin]

If you add one more json line to the import file to create a temp version, it will automatically create an export. Would that meet the need, rather than implementing a feature?

Sure, assuming the export only gets generated if the import was successful. What's that line?

[paynejd]

Example:

{"description": "Automatically generated repository version", "collection": "HTS-TST-N-MOH-Age-Sex-Result-NpsmKYKl0mt", "released": true, "owner": "DATIM-MOH-BI-FY19", "owner_type": "Organization", "type": "Collection Version", "id": "FY19.v0"}

[jamlung-ri]

CSV Example:

owner_id,collection,resource_type,id,description
WHO-Smart-Guidelines,IMMZ.C1.DE5-values,Collection Version,0.1.0,Initial Load

[snyaggarwal]

For Source:

{"type": "Source Version", "id": "v1.0", "source": "DemoSource", "description": "First release", "released": true, "owner": "DemoOrg", "owner_type": "Organization"}

[bmamlin]

The trick in my case is that I'm creating an export for HEAD. I don't create a new version until it has been validated.

[bmamlin]

Ok. After discussion on today's architecture call, we agreed that a more scaleable approach for this would be to trigger a new unreleased version using an instruction at the end of the import like @paynejd suggested.

I'll plan on adding a line containing JSON like this to future CIEL imports:

{ "type": "Source Version",
  "source": "CIEL",
  "id": "vYYYY-MM-DD",
  "description": "vYYYY-MM-DD",
  "released": false,
  "owner": "CIEL",
  "owner_type": "Organization"
}

Instead of validating an export of HEAD, I'll validate an export of the version and, if all looks good, simply mark it as released. If there are issues, I can delete the version and repeat the import.

@snyaggarwal, if the bulk import is split up and run in parallel jobs, is there a chance the new version could get triggered before all of the other resources have been processed? For example, parallel workers happily adding concepts & mappings in parallel, and then one of the workers finds this version instruction and triggers a new version while a couple of its sibling are only halfway through their resources and we end up with a version missing a few dozen concepts or mappings?