PAB not syncing/integration test melts RAM

lemmon-714 commented 2 years ago

Summary

Following the instructions here: https://github.com/input-output-hk/plutus-apps/tree/main/plutus-pab/test-node ...yields two unexpected behaviours, depending on how you look at it: Either PAB does not display expected behaviour (finish syncing*), or integration test crashes it.

*Hunch: PAB does not need to sync, it just fails at displaying things, because something is not connected properly.

Steps to reproduce the behavior

Follow https://github.com/input-output-hk/plutus-apps/tree/main/plutus-pab/test-node. I am also restarting all the steps often enough, and nothing else is happening on that machine.

Actual Result

A) At step 8: PAB is not syncing; meaning: Instead of

...
Current block: 327295. Current slot: 41749136.
Current block: 327295. Current slot: 41749150.

it is stuck at Starting PAB backend server on port 9080 for hours without change.

B) At step 9: Running the integration test without waiting for the PAB to sync increases RAM-usage until it crashes (15GB RAM + 64GB swap), without printing "Tx confirmed. Integration test complete."

Expected Result

PAB printing "Tx confirmed. Integration test complete." without crashing.

Describe the approach you would take to fix this

No response

System info

Ubuntu 20.04.3 LTS
almost fresh instance; not much else happened before on that machine
chain-index is synced
node is running
wallet-backend is running

lemmon-714 commented 2 years ago

One thought I had is that my solution here did not actually fix the issue, just the error message, and this some kind of resulting misbehaviour. Although it does appear that 'msc' has been just renamed to 'psc'.

ghost commented 2 years ago

Could you please try to synchronize with very small history by setting --rollback-history 1 option?

lemmon-714 commented 2 years ago

Could you please try to synchronize with very small history by setting --rollback-history 1 option?

Thank you, I will. Which part are you referring to, though? Node, chain-index, wallet-backend or PAB?

ghost commented 2 years ago

That's PAB's CLI flag.

lemmon-714 commented 2 years ago

Hi,

Update from my side.

Yesterday - before your last message - I restarted everything (except the integration test) and just let it run, in the hopes that was just being impatient. This morning however I found that PAB still kills itself without any message exceeding Starting PAB backend server on port 9080 (and a 'killed' of course).
Then tried your advice ~3.5h ago, after everything else was fully synced. Still stuck at

[pab:Info:11] [2022-01-26 09:43:13.76 UTC] Restoring PAB state ...
[pab:Info:11] [2022-01-26 09:43:13.76 UTC] PAB state restored with 6 contract instances.
[pab:Info:11] [2022-01-26 09:43:13.76 UTC] Starting PAB backend server on port 9080

RAM usage is at around 15GB still.

A number of questions.

Is this to be expected?
Could the issue be the 6 restored contract instances?
How can I kill them?

Thank you and best

lemmon-714 commented 2 years ago

Update from my side. 21h later no change (at least no crash). Turning off the instance now and delegating the task. EDIT: I lied, I did leave the instance running.

lemmon-714 commented 2 years ago

Update: After several days it shows one of those messages, in particular Current block: 100000. Current slot: 27243687

This happened after I started the integration test, and consumes about 15g RAM + 15g swap. Also all eight cores are under heavy load.

ghost commented 2 years ago

The situation with memory should be better since #315.

How can I kill them?

I think it's possible to delete the rows from the PAB's sqlite database with sqlite pab-core.db.

IntersectMBO / plutus-apps