search

lvgl / lv_binding_micropython

LVGL binding for MicroPython
MIT License
237 stars 156 forks source link

Move the Online Simulator backend away from Heroku #232

Closed amirgon closed 1 year ago

amirgon commented 1 year ago

Today we use SnippetBin as the backend for persisting Micropython scripts in the online simulator (the "Save" button), and it's hosted on Heroku Free Tier.

A few days ago I received this email form Heroku:

Dear Customer,

Thank you for being a Heroku user. Starting November 28, 2022, free Heroku Dynos, free Heroku Postgres, and free Heroku Data for Redis® will no longer be available. You can learn more about these and other important changes from our GM, Bob Wise, on the Heroku blog.

Existing free dynos and Heroku data add-ons will be impacted, so action by you is required. To prevent any disruption to your apps or data using free plans, you will need to upgrade from a free plan to a paid plan before November 28, 2022.

For instructions on how to upgrade and for other questions, please visit our FAQ.

Thank you,

The Heroku Team

We should decide how to proceed.

Options are:

@embeddedt , @kisvegabor - What do you think?

embeddedt commented 1 year ago

Heroku has been rock solid for this and other projects I've been involved in, so I'm not opposed to contacting them to see if we qualify for the open source program, assuming it's free. If it's not free we would need to compare with other options.

amirgon commented 1 year ago

I'm not opposed to contacting them to see if we qualify for the open source program, assuming it's free.

Even if it's free and available for us, I would think twice. I've read too many "Heroku is dying" titles in the last few years (this for example) and now removing the free tier makes me feel that Heroku will not be with us for long.

Why not simply host it on LVGL server? That way we don't depend on an external service.

embeddedt commented 1 year ago

I agree but it seems like a waste to pay for a fresh server just to run this.

@kisvegabor Does the VPS for the forum have any spare cycles or can it only run Discourse?

kisvegabor commented 1 year ago

Does the VPS for the forum have any spare cycles or can it only run Discourse?

As the Micropython simulator will use the server only to store store snippets I believe it won't overload the VPS. However, the Forum runs in Docker, and I don't really know how the port redirection are working and how can we serve two pages/services. It's possible for sure but requires some expertise.

We also need to use some kind of authentication to be sure only the simulator will store snippets there.

amirgon commented 1 year ago

the Forum runs in Docker, and I don't really know how the port redirection are working and how can we serve two pages/services.

Port redirection in Docker is pretty simple with the -p port:port option where you map the "inner" port to an "outer" port.

We also need to use some kind of authentication to be sure only the simulator will store snippets there.

This is a delicate point. I think it's possible to configure a single origin with CORS, so this provides some protection. Furthermore, we can limit the container storage such that in the worse case it would impact the simulator but not other services that are running on the same host.

If you think this is a good direction, I can try creating a Dockerfile to host SnippetBin.

kisvegabor commented 1 year ago

Port redirection in Docker is pretty simple with the -p port:port option where you map the "inner" port to an "outer" port.

I'm aware of that but I'm not sure it will be that simple :slightly_smiling_face:

Anyway, if someone is interested in implementing it I can provide access to the Forum's VPS.

amirgon commented 1 year ago

I don't have much experience with web server administration.... but I can give it a shot.

How is the domain sim.lvgl.io routed today?
Do you have some load balancer that could forward connections to sim.lvgl.io to some dedicated port on the Forum's VPS?

kisvegabor commented 1 year ago

How is the domain sim.lvgl.io routed today?

It's a GitHub feature that allows using a custom domain for GitHub pages. See here.

Do you have some load balancer that could forward connections to sim.lvgl.io to some dedicated port on the Forum's VPS?

We have no control over the sim.lvgl.io. It's managed by GitHub entirely.

embeddedt commented 1 year ago

I think we will need to use a second subdomain (snippetbin.lvgl.io) and possibly add a CORS header to allow sim.lvgl.io to access it.

kisvegabor commented 1 year ago

I think we will need to use a second subdomain (snippetbin.lvgl.io) and possibly add a CORS header to allow sim.lvgl.io to access it.

It's really not my territory to tell how to do this. However if you need me to do change some DNS or other settings, just let me know.

amirgon commented 1 year ago

@embeddedt if we simply set snippetbin_url to the forum's VPS with a special port, won't it simply work?
Then we can spin up a nodejs service (probably in a docker container) on the VPS, that listens to this port. Today CORS is permissive, so we can limit it to the VPS url to be more strict.

Am I missing anything?

embeddedt commented 1 year ago

Yes, this should work. Are you suggesting something like https://forum.lvgl.io:3000? It works but is a little confusing with forum still in the name.

amirgon commented 1 year ago

I would like to make progress in moving away from Heroku to LVGL VPS.
I created a Dockerfile that can be used like this:

sudo docker build -t snippetbin ./
sudo docker run -it --rm --name snippetbin -p 3000:3000 -e "SNIPPETBIN_DEPLOY_KEY=$(cat ~/SNIPPETBIN_DEPLOY_KEY.secret)" -e "SNIPPETBIN_REMOTE_REPO=amirgon/snippetbin_private_data.git" snippetbin npm start

I tested it locally and also fixed a few bugs on SnippetBin.

The environment variables SNIPPETBIN_REMOTE_REPO and SNIPPETBIN_DEPLOY_KEY specify the remote data repo for Snippetbin, which is currently a private repo under my username.
SNIPPETBIN_DEPLOY_KEY is a secret ssh key which is needed by SnippertBin for pulling and pushing to that data repo.

I suggest these next steps:

Optionally we can also:

(We can do the optional steps as a second phase after the deployment to the VPS)

@embeddedt @kisvegabor What do you think?

kisvegabor commented 1 year ago

Sorry, but I have too limited experience with these technologies to judge what would be the right direction. :slightly_frowning_face:

From administration perspective I'm perfectly fine with having a new subdomain or even a new VPS if required.

amirgon commented 1 year ago

Sorry, but I have too limited experience with these technologies to judge what would be the right direction.

From my side - I just need a "go" indication from you, or comments if you have any. Then I'll need your help with things I don't have access to (like creating a new LVGL data repo, VPS access etc.)

kisvegabor commented 1 year ago

I just need a "go" indication from you, or comments if you have any.

Then go :slightly_smiling_face: Thank you!

embeddedt commented 1 year ago

I think I have the necessary privileges to create any private repos you need but Gabor will have to help with access to the VPS.

kisvegabor commented 1 year ago

@amirgon I can send you the credentials of the VPS in private. Are you ok with that?

amirgon commented 1 year ago

I think I have the necessary privileges to create any private repos you need but Gabor will have to help with access to the VPS.

Could you create an LVGL private empty repo (snippetbin_data) and add me as a maintainer? I would need enough access rights to be able to create an SSH key, or you can create it instead. (I think you have to be an admin to create SSH keys)

I can send you the credentials of the VPS in private. Are you ok with that?

Sure, I'll try not to break anything :wink: Could you also create a new subdomain snippetbin.lvgl.io?

btw, do you know how the forum server is started today on the VPS? What starts it after the VPS reboots?

amirgon commented 1 year ago

I noticed that the VPS has only 4GB free out of 45GB disk, and already uses 1.6GB RAM out of a total of 2GB RAM.
I think it's a little... marginal... don't you think? 

Filesystem      Size  Used Avail Use% Mounted on  
/dev/vda1        49G   45G  4.2G  92% /      
              total        used        free      shared  buff/cache   available                                                                                                        
Mem:           1.9G        1.6G         92M        160M        250M         63M

The Discourse Docker running on the VPS limits RAM to 2GB which is all the available RAM, so it could easily consume all RAM and not leave resources for snippetbin:

CONTAINER ID        NAME                CPU %               MEM USAGE / LIMIT     MEM %               NET I/O             BLOCK I/O           PIDS     
dc9008a5d27f        app                 2.11%               1.313GiB / 1.947GiB   67.43%              121GB / 529GB       24TB / 8.68TB       151

For snippetbin, the nodejs docker image size alone is ~1GB, before counting the data size consumed by the container itself (snippetbin_data size mostly)

@kisvegabor - Is it an option to increase the VPS system resources a little?

embeddedt commented 1 year ago

Hmm... what's using all that space? It's only been 2 years; I'm skeptical that the forum data consumes 40 gigabytes already... but I could be wrong!

amirgon commented 1 year ago

Hmm... what's using all that space?

I'm mostly worried about RAM, but regarding disk space here is some analysis:

docker system prune helped a little to free 5GB. Looks like there are multiple versions of discourse/base images, I'm not sure why all of them are needed (each consumes ~2GB) 

Images space usage:           

REPOSITORY            TAG                 IMAGE ID            CREATED             SIZE                SHARED SIZE         UNIQUE SIZE         CONTAINERS
local_discourse/app   latest              1852c47936f3        16 months ago       2.81GB              2.234GB             576.5MB             1
discourse/base        2.0.20210415-1332   30e4746e631e        17 months ago       2.234GB             2.234GB             0B                  0
discourse/base        2.0.20200512-1735   991acdba0b1f        2 years ago         2.218GB             0B                  2.218GB             0
discourse/base        2.0.20190906-0522   6da16759d83d        3 years ago         2.293GB             0B                  2.293GB             0
discourse/base        2.0.20190625-0946   2b3a5b47565f        3 years ago         1.928GB             0B                  1.928GB             0
discourse/base        2.0.20190505-2322   ed87227f60d2        3 years ago         1.907GB             0B                  1.907GB             0

Containers space usage:

CONTAINER ID        IMAGE                 COMMAND             LOCAL VOLUMES       SIZE                CREATED             STATUS              NAMES
dc9008a5d27f        local_discourse/app   "/sbin/boot"        0                   542MB               16 months ago       Up 16 months        app

As to the rest:

Looking into Discourse data (du /var/discourse/shared -h --max-depth=2):

23M     /var/discourse/shared/standalone/letsencrypt
118M    /var/discourse/shared/standalone/redis_data
56K     /var/discourse/shared/standalone/ssl
277M    /var/discourse/shared/standalone/log
28K     /var/discourse/shared/standalone/state
2.7G    /var/discourse/shared/standalone/postgres_data
1.6G    /var/discourse/shared/standalone/postgres_data_old
6.7G    /var/discourse/shared/standalone/backups
16K     /var/discourse/shared/standalone/tmp
4.0K    /var/discourse/shared/standalone/postgres_backup
521M    /var/discourse/shared/standalone/postgres_data_older
2.1G    /var/discourse/shared/standalone/uploads
264K    /var/discourse/shared/standalone/postgres_run
14G     /var/discourse/shared/standalone
14G     /var/discourse/shared
embeddedt commented 1 year ago

docker system prune helped a little to free 5GB.

That should avert the issue for now. Perhaps we can also look into pruning some Discourse backups later (assuming there is more than one in the backups folder).

For snippetbin, the nodejs docker image size alone is ~1GB

This seems high; let me see if I can cut it down.

The Discourse Docker running on the VPS limits RAM to 2GB which is all the available RAM, so it could easily consume all RAM and not leave resources for snippetbin

According to some brief research Discourse indeed recommends using at least 2GB of RAM. However, it sounds like it shouldn't regularly need the full 2GB, so do we have any swap configured?

amirgon commented 1 year ago

According to some brief research Discourse indeed recommends using at least 2GB of RAM. However, it sounds like it shouldn't regularly need the full 2GB, so do we have any swap configured?

Discourse currently uses 1.3GB RAM. Swap is configure to 2GB.

I still think it would be best to increase the VPS resources a little, if possible.

kisvegabor commented 1 year ago

Is it an option to increase the VPS system resources a little?

Now we have this config: image

We can switch to this: image

Almost double everything for +12 USD (double price)

The next option is: image

Which seems overkill now.

I've removed all but the last backup. It should free ~6GB.

amirgon commented 1 year ago

Almost double everything for +12 USD (double price)

I see.
Let's see if we can work with current config.

We now have 17GB free disk space, should be enough for now. I'm more worried about RAM:

root@discourse-s-1vcpu-2gb-fra1-01:~# free -h
              total        used        free      shared  buff/cache   available
Mem:           1.9G        1.6G         72M        148M        258M         56M
Swap:          2.0G        1.0G        995M

That's 2GB RAM and 1GB of swap already in use. That doesn't leave much spares. Looking at memory usage per process:

  PID USER      PR  NI    VIRT    RES    SHR S %CPU %MEM     TIME+ COMMAND                                                                                                                                
 2032 1000      25   5 9500112 375772   7772 S 54.5 18.4   5017:01 ruby                                                                                                                                   
 1718 lxd       20   0  357064 213556 211160 S  5.9 10.5   0:17.47 postmaster                                                                                                                             
25628 1000      20   0 9032120 210024      4 D  1.3 10.3 281:20.43 ruby                                                                                                                                   
30766 1000      20   0 8987064 197764   3780 R  2.0  9.7 198:23.15 ruby                                                                                                                                   
30495 1000      20   0 8982888 186712   1288 S  1.0  9.1  88:25.44 ruby                                                                                                                                   
 1709 lxd       20   0  357200 165888 162464 S  5.6  8.1   0:03.87 postmaster                                                                                                                             
13864 1000      20   0 8962408 154400   1160 S  1.0  7.6  31:09.68 ruby                                                                                                                                   
 1968 lxd       20   0  352948 125620 125400 S  0.3  6.2  30:45.39 postmaster                                                                                                                             
 1973 1000      20   0  495820  92352   1336 S  0.7  4.5 135:43.33 ruby                                                                                                                                   
 2073 lxd       20   0  360552  90780  85392 S  0.0  4.4   0:00.23 postmaster                                                                                                                             
 1786 lxd       20   0  361752  60500  53648 S  0.0  3.0   0:00.48 postmaster                                                                                                                             
 1907 lxd       20   0  357040  58512  54508 S  0.0  2.9   0:00.38 postmaster                                                                                                                             
 1281 lxd       20   0  360736  54688  49248 S  0.3  2.7   0:00.93 postmaster                                                                                                                             
 1717 lxd       20   0  357236  49412  45704 S  1.0  2.4   0:01.42 postmaster                                                                                                                             
 1720 lxd       20   0  360844  47776  43984 S  1.3  2.3   0:01.50 postmaster                                                                                                                             
 1943 uuidd     20   0  470980  47744   1092 S  2.0  2.3   6450:46 redis-server                                                                                                                           
 2026 lxd       20   0  357256  45900  41308 S  0.0  2.2   0:00.12 postmaster                                                                                                                             
 1843 lxd       20   0  356236  42496  40232 S  0.7  2.1   0:01.36 postmaster                                                                                                                             
 2021 lxd       20   0  359604  37132  32056 S  0.0  1.8   0:00.14 postmaster                                                                                                                             
 1967 lxd       20   0  353484  28072  28020 S  0.0  1.4  87:00.38 postmaster                                                                                                                             
 2152 lxd       20   0  355120  25100  22456 S  0.0  1.2   0:00.02 postmaster                                                                                                                             
 2128 lxd       20   0  354700  22764  20572 S  0.0  1.1   0:00.03 postmaster                                                                                                                             
 2133 lxd       20   0  354700  17240  15280 S  0.0  0.8   0:00.03 postmaster

It really looks like most of this memory is consumed by Discourse (Discourse is running on Ruby, postmaster is postgresQL database).

Do you know how to reduce/limit Discourse memory consumption?

amirgon commented 1 year ago

Update - I've started a snippetbin service on the VPS. It works when using an IP (http://139.59.153.8:3000/) but not when using the domain name (http://forum.lvgl.io:3000/).

I'm not sure how domain names are managed, but it looks like not with DigitalOcean:

image

@kisvegabor - how should I configure the domain?

kisvegabor commented 1 year ago

That's 2GB RAM and 1GB of swap already in use. That doesn't leave much spares. Looking at memory usage per process:

We can switch to the 24USD option. +12 USD/month is ok.

I'm not sure how domain names are managed, but it looks like not with DigitalOcean:

This is how it builds up:

Now we have this DNS setting for the forum: image AFAIK, it just forward the forum subdomain to an IP and there are no port specific things here.

I disabled CloudFlare caching but http://forum.lvgl.io:3000/ is still not working. I'll ask our system admin.

kisvegabor commented 1 year ago

I asked the system admin and the short answer is "It's not that simple".

We concluded that it'd be better to have a new VPS instead of extending the current one. It doesn't matter if we pay 2x12 USD or 1x24USD but 2 VPS would be way simpler. Do you agree?

amirgon commented 1 year ago

I asked the system admin and the short answer is "It's not that simple".

Strange. Port forwarding should be simple.
Could you ask your system admin to elaborate?

We concluded that it'd be better to have a new VPS instead of extending the current one. It doesn't matter if we pay 2x12 USD or 1x24USD but 2 VPS would be way simpler. Do you agree?

Sure, but I think it's a waste of 12USD a month.
I still think we can use the existing VPS in its current configuration. SnippetBin memory usage is so small that it would still fit.

kisvegabor commented 1 year ago

He said it won't work because HTTPS communicates only on 443 port. We need "mod-proxy".

He also linked these 2 resources:

And finally we can have https://forum.lvgl.io/snippetbin

However, as the forum uses Docker all these might be more complicated.

embeddedt commented 1 year ago

@kisvegabor Shouldn't it be possible to set up an Apache/Nginx proxy that selects which server to proxy requests to based on the hostname? I know that this is possible for some TCP services using SRV records. E.g. if the request is sent to forum.lvgl.io, send it to Discourse; if it's sent to snippetbin.lvgl.io, forward it to SnippetBin instead. The DNS records would be set up so that forum and snippetbin point to the same IP.

If that's not possible and we have to buy a new VPS, we should configure and choose the domain names in such a way that it can be used for other services too, not just SnippetBin. Otherwise we will be in the same position when a third potential service comes along.

amirgon commented 1 year ago

He said it won't work because HTTPS communicates only on 443 port. We need "mod-proxy".

If SSL is the problem, I can change SnippetBin to work with HTTPS instead of HTTP (still on a different port, to live together with Discourse on the same host).

Would that help?

amirgon commented 1 year ago

@kisvegabor if configuring the domain is such a big problem - there is another option. Is 139.59.153.8 a static IP?
If you can guarantee this IP doesn't change, we can use it directly in the simulator. It's used internally and is not exposed to the users, so we don't need to have a pretty domain name there.

kisvegabor commented 1 year ago

Yes, 139.59.153.8 is static. So it seems ok for me to use it directly.

It's used internally and is not exposed to the users, so we don't need to have a pretty domain name there.

It's just exposed it here :smile: But I think it's not a big secret. Anyone can check which request goes to which IP. E.g. this is from Chrome's Network inspector for a random LVGL GitHub issue site for a random file image

kisvegabor commented 1 year ago

And as I sent my comment I realized that by "expose" you meant "the user don't have to deal with it". :slightly_smiling_face:

amirgon commented 1 year ago

I think we have a problem.
I've tried migrating to the new SnippetBin service on the VPS - but I get this error: 

spread.js:25 Mixed Content: The page at 'https://sim.lvgl.io/v9.0/micropython/ports/javascript/index.html?script_direct=7f13d85cb1954e2ddfe2c209b997d166638be763' was loaded over HTTPS, but requested an insecure XMLHttpRequest endpoint 'http://139.59.153.8:3000/load_file/7f13d85cb1954e2ddfe2c209b997d166638be763'. This request has been blocked; the content must be served over HTTPS.

The problem here is that the Simulator is served over HTTPS, but SnippetBin is served over HTTP.
Browsers block HTTP requests for pages served over HTTPS for security reasons.

I can change SnippetBin to work over HTTPS but I would need a certificate signed by a Certificate Authority (CA).
I think that a self-signed certificate is also blocked by the browser, so we would need it signed by a CA.

Some ideas:

@kisvegabor @embeddedt - What do you think?

kisvegabor commented 1 year ago

Serve SnippetBin over HTTPS with a new certificate. I'm not sure about the procedure to generate and sign it

Can we use Let's encrypt for it? I used it for the forum too (although most of the magic happened in Discourse).

amirgon commented 1 year ago

Can we use Let's encrypt for it? I used it for the forum too (although most of the magic happened in Discourse).

The browser checks that the domain name matches the certificate name, so with HTTPS we'll need to use a domain name and not IP. I can use your existing Discourse Let's Encrypt certificate, but the registered domain there is forum.lvgl.io and I can't use it because of the problem above...

kisvegabor commented 1 year ago

I see. Having an other VPS? :slightly_smiling_face:

We can use that other VPS for other purposes later too as we will have more control there if we need to deal with only our own simpler "modules" (and not the complex Discourse).

amirgon commented 1 year ago

I see. Having an other VPS?

Before giving up - I feel I totally don't understand something: Why is it so hard to have two services on the same VPS? This should be easy!
Both are HTTPS, served on different ports.

I'll try to summarize the key point I'm missing here:

The disadvantage of another VPS is not only the money. It means there is another machine to maintain update and reboot. It means there is another certificate that sometimes expires and need to be renewed etc.

Update

Cloudflare doesn't forward port 3000, but it does forward port 8443!

With this "magical" port number, this now works! https://forum.lvgl.io:8443/ It's not scalable, though, because Cloudflare forwards only specific ports.

kisvegabor commented 1 year ago

I've changed the Proxy policy of the Forum: image

Does it make any difference?

amirgon commented 1 year ago

Does it make any difference?

I'm not sure, but now with port 8443 it works! I've updated lv_micropython javascript branches to use https://forum.lvgl.io:8443 instead of heroku and as far as I can tell - everything works now.

Thanks for the help!
As the migration from Heroku to LVGL VPS is complete, I'm closing this.

embeddedt commented 1 year ago

@amirgon Just as a heads-up; I'm not sure how you configured your SSH key, but you may want to add it as a deploy key on lvgl/snippetbin_data instead of as your own personal key, as right now it seems that GitHub is attributing all the commits to your personal account (even though the author is different):

image

amirgon commented 1 year ago

as right now it seems that GitHub is attributing all the commits to your personal account (even though the author is different)

Strange. I'm using a deploy key.
Maybe it shows my name because I created the key?

kisvegabor commented 1 year ago

I'm not sure, but now with port 8443 it works!

I re-enabled the proxy then.