maige-app[bot]
commented
1 week ago Thank you for opening this discussion thread. This is a great space for technical scoping and brainstorming towards a potential proposal to merge GuardianConnector with Earth Defenders Toolkit (EDT-Offline or Kakawa).
Given the detailed nature of your post, it seems like a preliminary scoping and brainstorming session is the right approach. Feel free to continue adding thoughts and ideas as they come to you.
Anyone is welcome to contribute to the discussion. Let's keep the conversation going and see where it leads us. cc @luandro @iamjeffg
From Maige. How's my driving?
This is a thread for ongoing technical scoping towards a potential proposal to merge GuardianConnector with Earth Defenders Toolkit (EDT-Offline or Kakawa), given the known parallels and overlaps in objectives and users, albeit for distinct hosting environments currently. This thread is not intended to explore this from a conceptual or strategic perspective; rather, it's a space for us to dump and discuss ideas, concerns, blockers, and everything else from a software point of view.
With GuardianConnector, we are interested to explore moving to a "Docker Everything Bagel" model where we host the suite of tools (Superset, GuardianConnector Views, Frizzle, Filebrowser, Postgres, in the future Mapeo Cloud, Terrastories) as a multi-container Docker environment. The status quo of today is that we host each tool as an individual service (but shared VM) on Azure, with a relatively high degree of operational overhead and management, and at a high monthly cost. We will continue to want to host deployments online for our users in the future, but ideally much more streamlined (perhaps switching to PaaS hosting instead of Azure), and if this model can help us cut costs as well then that is a good thing.
With Kakawa, this model is already implemented through the use of balenaOS, which comes with a modified Docker daemon fork designed for managing device fleets with Balena Cloud. There are some specific nuances to hosting on balenaOS, but the core of service orchestration is within this docker-compose.yml file. Kakawa is deploying this service stack on Balena. Recently, @luandro and @rudokemper have experimented with adding a few GuardianConnector apps on this branch and the process seems pretty straightforward as it's largely all Docker compose config for services that already have published images on Dockerhub or elsewhere.
balenaOS is for a specific hosting environment and not the right tool for cloud deployment, so perhaps we could have an EDT Core from which both Kakawa and GuardianConnector could pull for specific hosting environments? Or there might be a better way to schematize how we can share and contribute to a common pool of resources.
Here is a list of what is currently offered in each toolkit. We can continue to edit this table as new things come to light.
terrastories-db)Mapeo-ts-bridgeandPataka-ts-bridgemapeo-bridge)edt-offline-portalpatakaNeedless to say, the goal is not to achieve a 1:1 overlap, as some services are designed for distinct hosting environments (online or offline). However, a shared stack could include: Postgres, Filebrowser, Portal, Frizzle, Superset, Mapeo Cloud, GCV, Terrastories (as a view), auth, and some shared infrastructure for storing structured and unstructured data.
Anyone is welcome to contribute to the discussion. We are not ready to actually work on this (nor have we formally given the green light for this to happen), so this is intended for preliminary scoping and brainstorming. I took some time to write this initial post as stage setting, but let's not set the bar for adding comments to be high; just dumping thoughts as they come to you is fine. Feel free to also correct any details that I got wrong. cc @luandro @iamjeffg