Revise versions of packages included in Fed-BioMed

[srcansiz]

In GitLab by @sharkovsky on Jan 9, 2023, 10:12

Revise versions of dependencies in Fed-BioMed:

• update conda environments .yaml files
• update node GUI yarn packages dependencies packages.json
• upgrade prerequisite docker-compose (v1) to docker compose (v2) https://docs.docker.com/compose/migrate/
• update docker container files (base image, dependencies)
• replace deprecated nosetests by pytest

General direction is to try to stick to up-to-date dependency versions (*). Exceptions:

• python 3.10 (because 3.11 is quite new and not yet supported by some dependencies)
• torch 1.x (because 2.0 is also new and not yet supported by some dependencies)
• boringtun 0.4.0 (because we could not have working setup with 0.5.x yet)

Tasks:

• manually test on systems supported (fedora, ubuntu, mac, windows)
  • [ ] Ubuntu 22.04 LTS will be tested with CI
  • [x] Fedora 38
  • [x] MacOS X 12.6.6
  • dev: 101, sklearn perceptron, medical folder dataset
  • vpn: 101, medical folder dataset (w/ batch size 4 and no local/global test, because 8GB RAM limit on test laptop)
  • [x] Windows 11 (21H2) w/ WSL2 Ubuntu 22.04 LTS
  • dev: 101, medical folder dataset, secagg
  • vpn: 101, gui, (not tested medical folder dataset)
• udpdate/test CI slaves
  • [x] Ubuntu 22.04 LTS
  • [ ] MacOS xxx not yet active
• [x] python version (3.9 to 3.10 or 3.11) => 3.10
  • [x] consequence: replace nosetests
• [x] conda envs packages update
  • include: fedbiomed-{node,network,researcher,gui} fedbiomed-ci ; exclude: fedbiomed-{doc,doc-test} (will change with github migration)
  • [x] check/adapt: conda env can be created
  • [x] check/adapt: pass unit tests
  • [x] check/adapt: run notebooks manually (tested on Fedora 38 for: 101, declearn, flamby, breakpoint, gpu, secagg, monai classification, mednist, opacus mnist, celeba, sklearn perceptron + medical folder dataset w/ 2 nodes and training plan approval)
  • [x] check/adapt: pass CI builds
• [x] gui packages
  • upgrade gui/ui/package.json content
  • check/adapt GUI launch & usage (add dataset, approve training plan, browse files, etc.)
• [x] docker-compose software (support docker compose plugin instead ? docker-compose standalone is not supported anymore, but need to check MacOS and Windows WSL2 cases)
• [x] docker images + packages in Dockerfile
  • [x] development mode restful
  • [x] all for VPN mode
  • boringtun upgrade => keep 0.4.0 for now, see comment below
  • nvidia cuda upgrade
  • wireguard-tools from package manager (simplify)
  • out of scope of this issue: we could explore replacing boringtun by wireguard or wireguard-go that are widely distributed via package managers
  • [x] test manually notebooks for VPN mode
  • done: 101, GPU, secured GUI, secure aggregation, medical folder dataset + training plan approval
• [x] update documentation for supported versions

Preliminary task:

• [ ] new CI slave with up to date system (see #486: need Fedora > 33 to use scikit-learn >= 1.2) see #531

Note: this issue was inspired by #293.

(*) in the future, we may want to support/package multiple versions of Fed-BioMed (one with up to date packages for secure/production deployments, one more laxist on package versions). This is beyond the scope of this issue.

[srcansiz]

In GitLab by @mvesin on May 24, 2023, 10:36

marked the checklist item check/adapt: run notebooks manually as completed

[srcansiz]

In GitLab by @mvesin on May 24, 2023, 14:34

marked the checklist item gui packages as completed

[srcansiz]

In GitLab by @mvesin on May 25, 2023, 09:42

marked the checklist item docker-compose software (support docker compose plugin instead ? docker-compose standalone is not supported anymore, but need to check MacOS and Windows WSL2 cases) as completed

[marc-vesin]

test dummy comment

[mvesin]

Unsuccessful tentative to use last version of boringtun v0.5.2. No problem with boringtun v0.4.0

• container builds correctly

  # install boringtun userspace implementation of wireguard
  # 
  # - match versions of debian & boringtun : up to date 0.5.2 is now ok for bullseye
  # - glitch: bullseye's apt-get cargo is too old vs boringtun's cargo packages dependencies, need to install
  #   up to date rust/cargo
  RUN curl https://sh.rustup.rs -sSf | sh -s -- -y && . /root/.cargo/env && \
     cargo install --locked --bin boringtun-cli --version ~0.5.2 boringtun-cli
  
  # install needed wireguard-tools
  RUN apt-get install -y wireguard-tools

• but wireguard setup fails when launching container.

  $ ${FEDBIOMED_DIR}/scripts/fedbiomed_vpn start mqtt
  ...
  Error response from daemon: Container 7d57946669718cc65c817c929bd183aeee94b4899f627726d847f062d6214824 is not running
  Key is not the correct length or format: `'
  CRITICAL: setting peer in wireguard interface failed with error : Command '['wg', 'set', 'wg0', 'peer', '', 'allowed-ips',  '10.220.0.2/32', 'preshared-key', '/dev/stdin']' returned non-zero exit status 1.

• investigating on a fedbiomed-vpn-vpnserver container & manually replaying wireguard commands show same issue (tested on Fedora 38 in fedbiomed-vpn-vpnserver container based on python:3.10-slim-bullseye ) See : https://www.wireguard.com/xplatform/ for doc of interface, or for example: https://github.com/cloudflare/boringtun/issues/163

$ WG_SUDO=1 boringtun-cli wg0
BoringTun started successfully

$ socat - UNIX-CONNECT:/var/run/wireguard/wg0.sock
get=1
listen_port=49683
errno=0

$ wg set wg0 private-key <(wg genkey)
$

# no error when setting key
$ socat - UNIX-CONNECT:/var/run/wireguard/wg0.sock
set=1
private_key=6040ad1d02595ad318d7b54673de74361d7cce0a1a617da5b851031243abe67a

errno=0

# but output does not seem to be coherent
$ socat - UNIX-CONNECT:/var/run/wireguard/wg0.sock
get=1
own_public_key=cf4ad568938b7535f70e7cc68c29e038b3022efb0fe5b9a6c9f76a7be5c71817
listen_port=49683
errno=0

$ wg show wg0
interface: wg0
  listening port: 49683

[mvesin]

Boringtun v0.5.2 unsuccessful tentative reported to boringtun developers : https://github.com/cloudflare/boringtun/issues/348