Add support for uploading/artifact management on Efsity

[ndegwamartin]

Overview

We need to add a new feature that supports app configs and fhir resources publishing and versioning

Implementation scope

New fct sub command publish

Flags

• -i --input project folder path (Must be a folder)
• -at --access-token the access token to use
• -bu --fhir-base-url the fhir base url
• -e --env (Optional) file path to the dot env properties file (Future scope)

FHIR Resources/Doc versioning

• Track the specific version of efsity that did the publishing
• Add meta data tag to all uploaded resources sample
  • System : https://smartregister.org/fct-release-version
  • Code: v2.3.0

Publishing state management

• Update project published version in local manifest
• Manifest file in hidden folder called .efsity/state.json
• Format: json
• Content:
  • Efsity version : efsity_version e.g. v2.3.0 from release tag
  • Last publish version: last_published_version (See section on versioning)
  • Timestamp created: created
  • An object array of:

    [
    {
    "path": "<project path relative to root>",
    "url": "<FHIR base url to send to>",
    "updated": "<Last updated timestamp ISO 8601 with timezone>"
    }
    ]

Note: :

• The url value should be unique in the array e.g. : http://fhir-stage.smartregister.org/fhir/
• The path is path to the folder in the context of the repo e.g. fhir-resources repo

Diff changes state management

• This will solve for issue on reposting all resources including unedited ones to the backend server
• Update file with the SHA1s of all files updated .efsity/publish_index.json
• The tool should only update meta tag and upload to the server files whose SHA1 is different
• After uploading each SHA1 entry is updated
• File format: json
• Content
  • A json file with key relative file path with resources repo
  • A value of the file’s SHA 1

Standard output/Logging

• Start and End time of the processing
• Total time/duration taken to run to completion
• Each file being processed with INFO level
• Success or Error status after server update e.g POST 200 OK http://url//fhir/x

Properties Management

• Dot env properties file format: .env
• Default location same as fct jar otherwise specify -e, --env flag
• Contains various properties configured as an alternative to passing CLI env variables or flags
• Configurable properties/env var keys
  • INPUT_FOLDER
  • ACCESS_TOKEN
  • FHIR_BASE_URL

[ndegwamartin]

Current draft of the Proposed FHIR Resources project structure referenced above

[ndegwamartin]

FHIR Resources managed in fhir-resources repository include:

• Composition
• Questionnaire
• StructureMap
• PlanDefinition
• Library
• Measure
• Binary (For app configs, translations - tool converts from actual json before upload.See https://build.fhir.org/binary.html)
• Observation (for supply chain commodities)
• Condition
• Basic
• List
• Group (for supply chain commodities)

[ndegwamartin]

Related conversation on versioning for the APK and handling Migrations https://www.notion.so/onaio/How-to-bundle-the-configuration-and-target-a-specific-app-version-0075da94ace04551b7ebde984ed975f2?pvs=4

[pld]

some thoughts:

• on 1 vs 2 digits, what is a breaking change in the context of content version? (if we can't define this deterministically, maybe we just use 1 digit for version? easier math, fewer decisions)
• we still need to tweak the folder structure
• for time, need to enforce including the TZ

[pld]

We made changes to the content structure, the latest is in this discussion and we can use for reference when implementing this, https://github.com/opensrp/fhircore/discussions/2787

[pld]

some thoughts:

* on 1 vs 2 digits, what is a breaking change in the context of content version? (if we can't define this deterministically, maybe we just use 1 digit for version? easier math, fewer decisions)

* we still need to tweak the folder structure

* for time, need to enforce including the TZ

@ndegwamartin can you give you're thoughts on this?

[ndegwamartin]

some thoughts:

* on 1 vs 2 digits, what is a breaking change in the context of content version? (if we can't define this deterministically, maybe we just use 1 digit for version? easier math, fewer decisions)

* we still need to tweak the folder structure

* for time, need to enforce including the TZ

@ndegwamartin can you give you're thoughts on this?

On the breaking changes, perhaps we can come up with some list of potentially breaking and non breaking changes e.g. if we remove an element that was existing before it is (most likely) breaking but if for instance we edit the Display, Text or Description elements the app logic will not necessarily break. Normally adding stuff would not break but we could add something like a sub element (some element within an element) and that breaks the app in some context.

We could provide some guidelines but this will ultimately be left to the discretion of the user since they'd be best placed to provide the judgment. In future though A.I. could help us out here 😀 .

[ndegwamartin]

@pld greed on the TZ, we can use the ISO 8601 variant with the Date + Time + TZ

[ndegwamartin]

On the versioning though, I think the main reason we needed it was to support APK runtime versioning. However if we go with the Manifest/Composition approach to tell the client what it can or cannot run gracefully then the versioning here would be redundant.

What may be more useful for us to do is to replace the versioning mentioned here with the Manifest/Composition/IG approach versioning meaning the versions update of the supported APK range would need to be made to those resources instead and in the new format. Notion ticket here

[pld]

OK, yes I think that's correct, I'm going to update the description to remove the version and include what we've agreed otherwise.

[pld]

hi @Wambere can you please take a look at this scope and let us know what unclear or needs further specification?

Note that the bottom section links to a v2 ticket that will not implemented in this issue, it's here so it can be taken into account the the v1/MVP version

[Wambere]

Diff changes state management

This will solve for issue on reposting all resources including unedited ones to the backend server Update file with the SHA1s of all files updated .efsity/publish_index.json The tool should only update meta tag and upload to the server files whose SHA1 is different After uploading each SHA1 entry is updated

It's technically not possible to re-post a resource with no changes, the server won't even accept it. So I don't see the value of having this functionality. Leaving it out until this is clarified