Shell escape

[martenlienen]

This PR fixes all escaping issues including the one from #50. The vscode configuration works as well as seen in the following example.

# experiment.yaml
seml:
  executable: 'matching_matrix.py'
  conda_environment: 'pytorch1.4'
  name: 'language_table'
  output_dir: '~/logs'
  project_root_dir: '.'

slurm:
  experiments_per_job: 1
  sbatch_options:
    mem: 32G
    gres: 'gpu:1'
    cpus-per-task: 2
    time: '0-1:00'

fixed:
  ntrain: 10000
  niter: 300
  lr_half_niter: 100
  print_niter: 999999999
  ninit: 2500
  lr: 500.
  dist_cluster: "l2"
  kernel: "l2"
  reg: 0.05
  p_prefix: "/nfs/staff-ssd/lienen/gdn-alignment-data/stanford-scans/p"
  language_src: "/nfs/staff-ssd/lienen/gdn-alignment-data/stanford-scans/armadillo.vec"
  language_tgt: "/nfs/staff-ssd/lienen/gdn-alignment-data/stanford-scans/dragon.vec"

deterministic:
  fixed:
    output_r: marten's data with "quotes" in quotes

$ seml <coll-name> print-command
********** First experiment **********
Executable: matching_matrix.py
Anaconda environment: pytorch1.4

Arguments for VS Code debugger:
["with", "--debug", "ntrain=10000", "niter=300", "lr_half_niter=100", "print_niter=999999999", "ninit=2500", "lr=500.0", "dist_cluster=\"l2\"", "kernel=\"l2\"", "reg=0.05", "p_prefix=\"/nfs/staff-ssd/lienen/gdn-alignment-data/stanford-scans/p\"", "language_src=\"/nfs/staff-ssd/lienen/gdn-alignment-data/stanford-scans/armadillo.vec\"", "language_tgt=\"/nfs/staff-ssd/lienen/gdn-alignment-data/stanford-scans/dragon.vec\"", "output_r=\"marten's data with \\\"quotes\\\" in quotes\"", "db_collection=\"stanford-scans3\"", "--unobserved"]
Arguments for PyCharm debugger:
with --debug ntrain=10000 niter=300 lr_half_niter=100 print_niter=999999999 ninit=2500 lr=500.0 'dist_cluster="l2"' 'kernel="l2"' reg=0.05 'p_prefix="/nfs/staff-ssd/lienen/gdn-alignment-data/stanford-scans/p"' 'language_src="/nfs/staff-ssd/lienen/gdn-alignment-data/stanford-scans/armadillo.vec"' 'language_tgt="/nfs/staff-ssd/lienen/gdn-alignment-data/stanford-scans/dragon.vec"' 'output_r="marten'"'"'s data with \"quotes\" in quotes"' 'db_collection="stanford-scans3"' --unobserved

Command for post-mortem debugging:
python matching_matrix.py with ntrain=10000 niter=300 lr_half_niter=100 print_niter=999999999 ninit=2500 lr=500.0 'dist_cluster="l2"' 'kernel="l2"' reg=0.05 'p_prefix="/nfs/staff-ssd/lienen/gdn-alignment-data/stanford-scans/p"' 'language_src="/nfs/staff-ssd/lienen/gdn-alignment-data/stanford-scans/armadillo.vec"' 'language_tgt="/nfs/staff-ssd/lienen/gdn-alignment-data/stanford-scans/dragon.vec"' 'output_r="marten'"'"'s data with \"quotes\" in quotes"' 'db_collection="stanford-scans3"' --unobserved --pdb

Command for remote debugging:
python -m debugpy --listen 172.24.64.16:40421 --wait-for-client matching_matrix.py with ntrain=10000 niter=300 lr_half_niter=100 print_niter=999999999 ninit=2500 lr=500.0 'dist_cluster="l2"' 'kernel="l2"' reg=0.05 'p_prefix="/nfs/staff-ssd/lienen/gdn-alignment-data/stanford-scans/p"' 'language_src="/nfs/staff-ssd/lienen/gdn-alignment-data/stanford-scans/armadillo.vec"' 'language_tgt="/nfs/staff-ssd/lienen/gdn-alignment-data/stanford-scans/dragon.vec"' 'output_r="marten'"'"'s data with \"quotes\" in quotes"' 'db_collection="stanford-scans3"' --unobserved

********** All raw commands **********
python matching_matrix.py with ntrain=10000 niter=300 lr_half_niter=100 print_niter=999999999 ninit=2500 lr=500.0 'dist_cluster="l2"' 'kernel="l2"' reg=0.05 'p_prefix="/nfs/staff-ssd/lienen/gdn-alignment-data/stanford-scans/p"' 'language_src="/nfs/staff-ssd/lienen/gdn-alignment-data/stanford-scans/armadillo.vec"' 'language_tgt="/nfs/staff-ssd/lienen/gdn-alignment-data/stanford-scans/dragon.vec"' 'output_r="marten'"'"'s data with \"quotes\" in quotes"' 'db_collection="stanford-scans3"' overwrite=1 --force

Putting the vscode command into a launch.json runs the following.

$ /usr/bin/env /home/ml/.pyenv/versions/3.9.6/bin/python /home/ml/.vscode/extensions/ms-python.python-2021.7.1060902895/pythonFiles/lib/python/debugpy/launcher 45547 -- <path>/run.py with --debug ntrain=10000 niter=300 lr_half_niter=100 print_niter=999999999 ninit=2500 lr=500.0 dist_cluster=\"l2\" kernel=\"l2\" reg=0.05 p_prefix=\"/nfs/staff-ssd/lienen/gdn-alignment-data/stanford-scans/p\" language_src=\"/nfs/staff-ssd/lienen/gdn-alignment-data/stanford-scans/armadillo.vec\" language_tgt=\"/nfs/staff-ssd/lienen/gdn-alignment-data/stanford-scans/dragon.vec\" "output_r=\"marten\'s data with \\\"quotes\\\" in quotes\"" db_collection=\"stanford-scans3\" --unobserved

[gasteigerjo]

Beautiful! :)

[n-gao]

This seems to cause some regression. Occasionally, dictionaries are now converted to strings with the dictionary as JSON as value. I had no time yet to investigate this properly but will do so in the coming days.

[gasteigerjo]

Oh, yes, that'd be great! We use recursion to convert dictionaries, so maybe this is missing here.