bug: backend crash when we hover the Edit button on the frontend UI

whollacsek commented 3 months ago

Describe the bug

This is a new instance I installed couple days ago and the only activities where sync'ing flows from another instance.

Whenever I hover the Edit button in a flow this happens:

Then I have to restart the container but somehow since this issue appeared the container doesn't start properly here is the log (notice the logs from src/monitor.rs:298 that are not printed in another healthy instance I have):

2024-06-24T16:45:27.237122Z  INFO src/main.rs:125: jemalloc enabled
2024-06-24T16:45:27.237147Z  INFO src/main.rs:157: Binary is in 'server' mode
2024-06-24T16:45:27.237172Z  INFO src/main.rs:250: Connecting to database...
2024-06-24T16:45:27.264046Z  INFO src/main.rs:252: Database connected
2024-06-24T16:45:27.268736Z  INFO src/main.rs:256: PostgreSQL version: PostgreSQL 16.3 (Debian 16.3-1.pgdg120+1) on x86_64-pc-linux-gnu, compiled by gcc (Debian 12.2.0-14) 12.2.0, 64-bit (windmill require PG >= 14)
2024-06-24T16:45:27.269352Z  INFO windmill-api/src/db.rs:78: Acquiring global PG lock for potential migration with pid: Some(1872)
2024-06-24T16:45:27.269833Z  INFO windmill-api/src/db.rs:98: Acquired global PG lock
2024-06-24T16:45:27.273272Z  INFO windmill-api/src/db.rs:113: Releasing PG lock
2024-06-24T16:45:27.273629Z  INFO windmill-api/src/db.rs:121: Released PG lock
2024-06-24T16:45:27.273664Z  INFO src/main.rs:294:
##############################
Windmill Community Edition v1.352.0-6-g189a2815a
##############################
2024-06-24T16:45:27.273691Z  INFO src/main.rs:651: config: MODE: server, GO_PATH: /usr/local/go/bin/go, PATH: /usr/local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/local/go/bin, HOME: /root
2024-06-24T16:45:27.278900Z  INFO windmill-common/src/worker.rs:99: Loaded setting custom_tags, common: ["chromium"], per-workspace: {}
2024-06-24T16:45:27.279352Z  WARN windmill-api/src/oauth2_ee.rs:180: oauth.json not found, no OAuth clients loaded
2024-06-24T16:45:27.280508Z  INFO src/monitor.rs:963: Reloading server config...
2024-06-24T16:45:27.280688Z  INFO src/monitor.rs:697: Loaded setting retention_period_secs from db config: Number(2592000)
2024-06-24T16:45:27.281041Z  INFO src/monitor.rs:671: Loaded saml_metadata setting to None
2024-06-24T16:45:27.281255Z  INFO src/monitor.rs:671: Loaded scim_token setting to None
2024-06-24T16:45:27.281875Z  INFO src/monitor.rs:298: 2.1177520751953125 mb allocated/11.92578125 mb resident
2024-06-24T16:45:27.321315Z  INFO windmill-api/src/embeddings.rs:222: Loading embedding model...
2024-06-24T16:45:27.321743Z  INFO src/main.rs:644: Successfully connected to pg listen
2024-06-24T16:45:27.321775Z  INFO windmill-api/src/embeddings.rs:196: Found embedding model in cache
2024-06-24T16:45:27.384087Z  INFO windmill-api/src/lib.rs:339: server started on port=8000 and addr=0.0.0.0 instance=IXGH7
2024-06-24T16:45:27.492744Z  INFO windmill-api/src/embeddings.rs:238: Loaded embedding model
2024-06-24T16:45:27.492769Z  INFO windmill-api/src/embeddings.rs:625: Creating embeddings DB...
2024-06-24T16:45:29.013147Z  INFO windmill-api/src/embeddings.rs:632: Created embeddings DB
2024-06-24T16:45:57.284491Z  INFO src/monitor.rs:298: 161.61193084716797 mb allocated/272.140625 mb resident
2024-06-24T16:46:27.286701Z  INFO src/monitor.rs:298: 161.98434448242188 mb allocated/272.359375 mb resident
2024-06-24T16:46:57.288199Z  INFO src/monitor.rs:298: 162.03257751464844 mb allocated/272.3359375 mb resident
2024-06-24T16:47:27.290323Z  INFO src/monitor.rs:298: 162.0587921142578 mb allocated/272.3359375 mb resident

To reproduce

Go to '...'
Click on '....'
Scroll down to '....'
See error

Expected behavior

No crashes

Screenshots

No response

Browser information

No response

Application version

CE v1.352.0-6-g189a2815a

Additional Context

No response

rubenfiszel commented 3 months ago

It's either a cache issue or a load balancer misconfiguration issue. Either your asset requests are not going to your servers or the cached frontend you have is not the same version that your server provide (in which case the request logs of your servers would show lots of 404)

whollacsek commented 3 months ago

In what case would the server change the asset version? I didn't even restart the container before this issue appeared.

On Mon, 24 Jun 2024 at 19:45, Ruben Fiszel @.***> wrote:

It's either a cache issue or a load balancer misconfiguration issue. Either your asset requests are not going to your servers or the cached frontend you have is not the same version that your server provide (in which case the request logs of your servers would show lots of 404)

— Reply to this email directly, view it on GitHub https://github.com/windmill-labs/windmill/issues/3963#issuecomment-2187093467, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAGJQUDNCAXUMZATK335KX3ZJBLJZAVCNFSM6AAAAABJ2GKMFKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCOBXGA4TGNBWG4 . You are receiving this because you authored the thread.Message ID: @.***>

rubenfiszel commented 3 months ago

I misread the issue, could you print the service logs of the server that crashed and the logs of the container that doesn't start properly

whollacsek commented 3 months ago

So I realized the backend didn't crash or at least it didn't produce any logs, is there another log besides the one I shared above?

rubenfiszel commented 3 months ago

I do not know your setup so yes possibly? Anyway, a request for an asset didn't reach your server when it should have which is why the frontend is displaying that error.

whollacsek commented 3 months ago

That's really strange with only this .css file that fails. Do you happen to know if there's any changes that I can try on Cloudflare? I've already added a Page rule to bypass the cache but it didn't help. In any case thank you for taking a look, I'll try to redeploy and see if the issue comes back

On Mon, Jun 24, 2024, 21:43 Ruben Fiszel @.***> wrote:

I do not know your setup so yes possibly? Anyway, a request for an asset didn't reach your server when it should have which is why the frontend is displaying that error.

— Reply to this email directly, view it on GitHub https://github.com/windmill-labs/windmill/issues/3963#issuecomment-2187281942, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAGJQUAZYKAHVDNF2HAWZTLZJBZHPAVCNFSM6AAAAABJ2GKMFKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCOBXGI4DCOJUGI . You are receiving this because you authored the thread.Message ID: @.***>

windmill-labs / windmill