Consider repo structure for future deployments

[edmoss345]

Need to potentially reorganise the repositories so that there is an easy way to reference global content and deployment specific content, and allow third parties to build chatbots with minimal supervision

[edmoss345]

@istride @fagiothree based on our discussion on 18/10, we were looking at two general structures as visualised below

There is already a 'parenttext-core' repo https://github.com/IDEMSInternational/parenttext-core/tree/main, that we created when we were doing some investigations into this structure before. I will repurpose this to do a proof of concept for the second approach above and then we can discuss next steps. At the very least parenttext-core should hold the template references and the models files.

[edmoss345]

https://github.com/IDEMSInternational/parenttext-core/pull/1

[istride]

In the diagram above, is "Chatbot-pipeline" essentially the same as our existing parenttext-pipeline, or is it something else?

In any case, I expect the deployment specific repo, if it contained all deployments, to look more like:

untitled-repo
├── china
│   ├── config.py
│   └── edits
│       ├── safeguarding_words.json
│       ├── select_phrases.json
│       ├── special_words.json
│       └── specific_expiration.json
├── core
│   └── models
│       └── parenttext_models.py
├── malaysia
│   ├── config.py
│   └── edits
│       ├── safeguarding_words.json
│       ├── select_phrases.json
│       ├── special_words.json
│       └── specific_expiration.json
├── mexico
│   ├── config.py
│   └── edits
│       ├── safeguarding_words.json
│       ├── select_phrases.json
│       ├── special_words.json
│       └── specific_expiration.json
├── package.json
├── package-lock.json
├── README.md
├── requirements.txt
└── south_africa
    ├── config.py
    └── edits
        ├── safeguarding_words.json
        ├── select_phrases.json
        ├── special_words.json
        └── specific_expiration.json

The contents of each deployment sub-folder may vary over time. Quite possibly, they would only contain a config.py to begin with, but separating each deployment in its own sub-folder gives the flexibility to change the contents independently.

[edmoss345]

Thanks Ian, the above makes good sense, agreed splitting into folders. I will do some general reorganisation ahead of our discussion tomorrow. In response to your first question, yes I'm envisioning the the parenttext-pipeline repo is the chatbot-pipeline repo. SO we pull anything parenttext specific out of there

[istride]

Blocked; waiting for higher level discussion about what our needs are regarding this, which should be concluded by 27 November 2023.

[istride]

After discussion, the following things are clear:

• Fundamentally, this is about the content of the spreadsheets being under version control
• We want to enable a workflow where development occurs at the right level e.g. in the deployment-specific repo for issues that affect only a single deployment; Github Issues and PRs may be useful for this and would work best with separate repositories
• We want to allow changes to flow upstream as well as downstream
• Changes in one repository must not be imposed on others - changes are pulled in by the repository owner
• We want all the features of a VCS like git to be available to manage changes at all levels e.g. branches, tags
• It is expected that spreadsheets will be converted to a universal (across App and Text) JSON format before being committed

It is anticipated that there will be several levels of content repositories:

chatbot              (for any chatbot)
└── parenttext       (for all ParentText chatbots)
    ├── malaysia     (for a specific deployment)
    └── south-africa

We should also consider how to decrease the maintenance burden of Github workflows across all these content repositories. One possibility is to keep all workflows in a single repository and have them be referenced by the deployment-specific repositories - see reusing workflows.

We should aim to move as much as possible into the spreadsheets i.e. config.py.

@fagiothree @ChrisMarsh82 please review and let me know if I have missed anything out, or misunderstood.

[edmoss345]

@istride @geoo89 @fagiothree slightly repurposing this issue as it is related to our recent discussion of how we enable quick deployments of chatbots. Have created a google doc for us to dump our ideas in for now.

https://docs.google.com/document/d/1rSKGf9XA5vcehoMkkd46K2F1hNVMD2xHrzQTU-IfgK4/edit?usp=drive_link