Rewriting Scripts in Rust tracking issue

[maan2003]

We have a bunch of integration tests written in bash. They are flaky, hard to write and update. We want to re-write all these tests in Rust and minimize the amount of bash we depend on.

We've made 2 PRs in this project so far:

• https://github.com/fedimint/fedimint/pull/1899 Added ability to spawn daemons (e.g. fedimintd and lnd etc) via Rust
• https://github.com/fedimint/fedimint/pull/1910 Can write tests purely in rust. As a proof-of-concept, it re-wrote latency-test.sh:
  • Old bash version that was replaced
  • New rust version which is called here

Work remaining (in order of importance):

• [ ] (1) Export $FM_LOGS_DIR in github actions. This helps us debug logs of specific daemons. See upload-artifact
• [x] Rewrite the rest of the bash integrations tests:
  • [x] (2a) scripts/cli-test.sh (easier)
  • [x] (2b) reconnect tests (harder b/c need to stop and start fedimintd)
  • [x] (2c) upgrade tests (harder b/c need to stop and start fedimintd, but 2b has now demonstrated how to do it)
• [x] (3) Replace scripts/setup-tests.sh with a rust-spawned dev federation. For example, something like this function. Perhaps rename to dev_federation or something. blocked on (2b) (2c)
• [x] (4) Spawn esplora in fedimint-bin-tests. This functionality exists in fedimint-testing/src/bin/fixtures.rs
• [x] (5) Spawn electrs in fedimint-bin-tests. This functionality exists in fedimint-testing/src/bin/fixtures.rs
• [x] (6) Don't set any environment variables in scripts/build.sh. Just run cargo build .... Instead, set environment variables in dev_fed Rust function.
• [x] (7) Once nothing depends on it, remove scripts/lib.sh
• [x] (8) Once nothing depends on it, remove fedimint-testing/src/bin/fixtures.rs
• [x] (9) https://github.com/fedimint/fedimint/issues/2137
• [ ] (10) Ability to run dkg with n peers and run fedimintd with fewer than n peers
• [x] (11) Ability to run fedimintd without running DKG beforehand. This will allow us to replace scripts/run-ui.sh which is useful to test setup UI
• [x] Use PATH instead of FM_BIN_DIR
• [x] When any cmd! fails, log the the entire command that was run
• [x] devimint should peg-in user and gateways for mprocs and tmux developer environments. Stop using pegin.sh and then delete it.
• [x] Write distributedgen to a log file. Probably the same one that fedimintd uses. This way errors will show up visually in mprocs.
• [ ] When devimint tmuxinator is running, we should have a command to start / stop the fedimintd servers.
• [x] include! daemon config files so external projects don't need them.
  • [ ] Eventually an option to override these default configs

If you'd like to tackle one of these issues, just post "I'll try (\<n>)". If multiple people want to try the same thing, try to collaborate in discord. Make draft pull requests early for feedback. @Maan2003 can help you!

[maan2003]

will re add good first issue ater latency tests are fixed.

[ayush-ranjan-official]

@Maan2003 I'll try (4)

[gotlougit]

I'd like to work on (5)

[GetPsyched]

I'd like to work on (2a)

[maan2003]

@gotlougit @ayush-ranjan-official (4) and (5) are very similar, feel free to collaborate and do it in one PR.

[oleonardolima]

@Maan2003 I'd like to work on (3)

[maan2003]

FYI you don't need to ping me to say that. You just need to say it here so that there is no duplicated work.

But feel free to ping me here or on discord if you get stuck somewhere.

[maan2003]

@oleonardolima updated description of (3)

[maan2003]

@oleonardolima that update was incorrect, Sorry. updated again

[debjit-bw]

I want to take up (6)

[ayush-ranjan-official]

@Maan2003 I'll try (4)

I guess (4) is solved so I'd try (1)

[oleonardolima]

@oleonardolima that update was incorrect, Sorry. updated again

No problem, as it's blocked on (2b) and (2c), I'll work on (2b) first.

[dhruv-1001]

Hey @oleonardolima I was working on (2b), sorry I forgot to mention it here and started working on it. Do you mind if I continue with (2b)? (I'm ready to push a draft PR)

[maan2003]

There is no harm in opening a draft PR. You should collaborate to get the rest of the work done.

[maan2003]

Added a reference for 1

[justinmoon]

Now that we've completed 2b, 2c should be easier because there are hooks to start and stop fedimintd

[dhruv-1001]

If no one is working on 2c, i'll take this one.

[justinmoon]

If no one is working on 2c, i'll take this one.

Go for it!

[jkitman]

@dhruv-1001 Don't worry about (2c) I'm going to test that one in another way (I'm changing how the upgrade API works).

[maan2003]

I am working (6).

[maan2003]

I am working on (3).

[maan2003]

devimint should peg-in user and gateways for mprocs and tmux developer environments. Stop using pegin.sh and then delete it.

we can't delete scripts/pegin.sh until we have replacement for it that can invoked from the user shell.

[justinmoon]

we can't delete scripts/pegin.sh until we have replacement for it that can invoked from the user shell.

Good point. Let's wait for that until @elsirion finishes the wallet commands for client-ng. That's what he's working on now.

[justinmoon]

But the user shell wouldn't need to do this. dev_fed could do this.

[techsavvyash]

i'll try (17 (include! daemon config files)

[justinmoon]

@techsavvyash: Great. Feel free to just make a draft PR once you've got it working with 1 daemon.

[justinmoon]

Closing this because the initial project is complete. Might add another issue with devimint improvement ideas.