search

finos / waltz

Enterprise Information Service
https://waltz.finos.org
Apache License 2.0
182 stars 129 forks source link

Waltz Public Instance #5906

Closed pedroleaoc closed 2 years ago

pedroleaoc commented 2 years ago

Description

The idea is to create a public instance of Charmed Waltz running on FINOS infrastructure. Similar to other FINOS public instances, members could explore the tool before devoting the time and resources to install it locally. The deployment would be made with Juju on Kubernetes following the work performed by Charming Waltz Working Group.

cc @mcleo-d @maoo

Resourcing

We would like to collaborate on this feature

davidwatkins73 commented 2 years ago

I fully support this proposal.

maoo commented 2 years ago

Thanks for the issue @pedroleaoc and @davidwatkins73 for confirming your support!

My understanding is that you'd like to have a public Waltz instance like the FINOS Legend Shared instance on https://legend.finos.org/studio

From a technical standpoint, what we'd need to figure out is:

I expect that costs will be manageable, but I'll have to submit the amount for review anyway.

Please let me know if I missed anything, and if you have any questions. Thanks!

mcleo-d commented 2 years ago

Link to sample data on @davidwatkins73 GitHub profile.

https://github.com/davidwatkins73/waltz-demo-docker/tree/master/waltz-sample-db

mcleo-d commented 2 years ago

Deployment mechanism is ready and tested (Juju charms for EKS + local, with microk8s)

Targeting AWS EKS has been confirmed by the Waltz community.

cc @davidwatkins73

pedroleaoc commented 2 years ago

About the cloud infra, what do you folks think about using a tinny EC2 instance with MicroK8s instead of EKS? As pointed out by @mcleo-d, we should also consider CFI.

mcleo-d commented 2 years ago

Hey @pedroleaoc - There shouldn't be an issue. I'd suggest engaging with @maoo.

maoo commented 2 years ago

I already discussed this with Pedro ; in fact, his proposal comes from the request to keep costs under control, given that this is a demo instance. Based on its usage, we will later decide if/when/how to scale it.

@pedroleaoc - let's use Ec2 + microk8s as reference architecture for Waltz. Thanks!

davidwatkins73 commented 2 years ago

Q: Authentication + authorization mechanism: will Waltz provide basic signup/signin ? Will access be restricted? Or there will be few pre-defined users/passwords that can be used to access the environment?

A: Waltz will allow for basic sign in (user + password). @davidwatkins73 to ensure the sample data has a set of users and passwords.

davidwatkins73 commented 2 years ago

Q: Data management: will data be reset periodically, ie on weekly bases? Will there be data sandboxes for each user access? Which part of data is preserved and which one is reset? @davidwatkins73 , @balbirthomas , @pedroleaoc

A: Data should be reset periodically (weekly?). No sandboxes per user. May need to update the client/server digest to force a reload after a database reset. Initially we will look to use juju to perform the reset.

davidwatkins73 commented 2 years ago

Q: Monitoring and Scaling

A: (Partial) Scalability should not be a problem, we run entire DB instance on a pair of 2gb (shared) tomcat services.

mcleo-d commented 2 years ago

A: (Partial) Scalability should not be a problem, we run entire DB instance on a pair of 2gb (shared) tomcat services.

@maoo - Regarding monitoring Tomcat, does AWS provide a notification service that could inform an email address if the services are interrupted or error?

maoo commented 2 years ago

A: (Partial) Scalability should not be a problem, we run entire DB instance on a pair of 2gb (shared) tomcat services.

@maoo - Regarding monitoring Tomcat, does AWS provide a notification service that could inform an email address if the services are interrupted or error?

We can use the https://uptimerobot.com/ FINOS account to point to a public health-check URL provided by Waltz, if available; if not available, we'll rely on the login page to be available.

claudiubelu commented 2 years ago

Apologies, it seems that something was up with the domain name that I own. I created a temporary domain name on another service and generated a new certificate. This is my current Waltz deployment on EKS + HTTPS enabled: https://test-waltz.ipq.co/

balbirthomas commented 2 years ago

Just keeping you updated @davidwatkins73 @mcleo-d . I am still exploring alternative solutions with multiple Juju/charm teams with regard to periodic resetting of Waltz public instance data. I will get back to you shortly on this, once the stars align :-) .

mcleo-d commented 2 years ago

Hi @kriscanonical, @balbirthomas, @pedroleaoc, @claudiubelu 👋🏻

FINOS is planning an in-person Waltz meetup for Wednesday 20th April in London to help educate the FINOS community on Waltz and help drive contribution.

Having the Waltz AWS public instance done and available to browse would be an awesome feature to demo at the event. Do you think the following milestones are achievable to get this done ...

I'm happy to setup a quick project team call to discuss if needed?

James.

ghost commented 2 years ago

Thanks @mcleo-d. We will plan to be there in-person for Waltz meetup in London.

Cheers, Kris

mcleo-d commented 2 years ago

Hi - I have just linked this issue to the FINOS London Meetup on 20th April that is Waltz themed. It would be great if this issue can be delivered in order to be demonstrated here https://github.com/finos/waltz/issues/5986

cc @maoo

claudiubelu commented 2 years ago

Hi - I have just linked this issue to the FINOS London Meetup on 20th April that is Waltz themed. It would be great if this issue can be delivered in order to be demonstrated here #5986

cc @maoo

Hello, currently waiting a reply on this issue: https://github.com/finos/waltz/issues/5966 , which is needed for the public instance to have the public data set.

mcleo-d commented 2 years ago

cc @maoo and @pedroleaoc,

robmoffat commented 2 years ago

SSL Cert instructions here: https://github.com/finos/waltz-juju-bundle/pull/1

mcleo-d commented 2 years ago

@TheJuanAndOnly99 and @robmoffat - Following the Waltz project meeting #6089, it's been determined SSL Certificate for the Waltz Public Instance can be applied by the FINOS infrastructure team.

https://github.com/finos/waltz/issues/5906#issuecomment-1097940997

SSL certificate on demo.waltz.finos.org - @TheJuanAndOnly99 and @robmoffat

It would be great to schedule to move this forward. Thanks!

James.

maoo commented 2 years ago

@mcleo-d - unfortunately there's not much we can do on this, until we get a stable and working automation from @pedroleaoc and team.

robmoffat commented 2 years ago

Just these last two steps to figure out:

maoo commented 2 years ago

Docs can be found on:

I think we can close this issue now, as the actions are in place, badges are available on https://github.com/finos/waltz-juju-bundle , the EC2 instance is working, with SSL and docs are in place.

@davidwatkins73 - feel free to read/improve docs, ask questions to @claudiubelu and team, link docs into the main Waltz readme and play a bit with GitHub Actions.

claudiubelu commented 2 years ago

Hello,

As you can see, the SSL certificate has been added to demo.waltz.finos.org. The Waltz Public Instance is currently running with the PostgreSQL sample database which can be found here: https://github.com/finos/waltz/blob/master/waltz-sample-data/database/pg/dump.sql.tar.gz . The sample database is being reset every 2 hours, as we've decided some time ago (any changes to the data will be reset).

Waltz can be deployed by using the finos-waltz-k8s charm (https://github.com/finos/waltz-integration-juju and https://charmhub.io/finos-waltz-k8s). It can be configured with an external PostgreSQL database, or it can be configured to run with a postgresql-k8s charm, which will create and maintain the PostgreSQL database instead (including credentials created for any related charm).

Typically, multiple charms can be deploy a lot easier through a bundle, and we do have a Waltz bundle, which can be found here: https://github.com/finos/waltz-juju-bundle and https://charmhub.io/finos-waltz-bundle?channel=1.39/edge . There is also a Waltz Public bundle, which will also contain the PostgreSQL sample database as well (can be found here: https://github.com/finos/waltz-juju-bundle/tree/public and https://charmhub.io/finos-waltz-bundle?channel=public/edge).

As for the documentation, there are a few noteworthy ones:

We have quite a few github actions running on these projects, including lint, unit test, and integration tests that are running on every Pull Request. There are a few actions that are using various tokens which have a limited TTL, which would have to be refreshed in the future. They are also documented here:

Finally, the Waltz Public Instance is updated daily to the latest revisions due to this action. For more information about how it was set up, see here: https://github.com/finos/waltz-integration-juju/blob/main/docs/periodic_job.md

If I missed anything, please let me know. :)

maoo commented 2 years ago

Love the recap @claudiubelu , thanks so much! I wonder if we could copy/paste this content somewhere? I see that docs are located in many places, so it may be interesting to understand "what is where" and if there are chances to consolidate them in some way, eager to hear your feedback.

Finally - slightly connected - there's an open issue that is currently preventing us to do automatic releases of the Juju bundle, see https://github.com/finos/waltz-juju-bundle/issues/15 . @pedroleaoc is trying to reproduce and follow up internally.