Clean the VM

[MarvinLvn]

Big spring fall clean ! What needs to be done :

• [x] Merge scripts tools, varia folders and any other ancillary wrappers that are used in multiple tools into utils
• [x] Create launcher folder to store all of the high-level scripts that launch some tool
• [x] Standardize launching scripts so that their name is always the tool name, and they always take in a folder, as well as the minimum absolutely necessary parameters (for SAD, nothing; for diarization, a SAD may be provided); and that they also accept a parameter (--keep-temp)
• [x] Create data/temporary folder where all the temporary data will be stored (if --keep-temp asked by the user; otherwise, this folder's contents get emptied after running)
• [x] Create repos folder, inside which there will be one folder per repo cloned into the VM (one per each sad, diarizer, etc)
• [ ] Inside each tool subfolder (e.g., tools/OpenSAT), there will be a wrapper that specifies the behavior to adopt regarding the files that need to be processed*
• [x] Get rid of all names that are confusing
• [x] Separate the docs into what minimally has to be there (how to install, build, uninstall; how to run the test.sh [and test-daylong.sh --- when this one is implemented]) and what will depend on what tools get installed correctly - only the former gets built WITH the VM
• [x] In the test files, incorporate a routine that harvests doc files from within each tool folder (eg. tools/OpenSAT should contain an .rst file) ONLY when the tool passed the test. (note: docs get rebuilt every time the user manually launches a test and only then.)

*notes regarding this intermediate wrapper (from a conversation between Eric and Alex):

• The job of this tool-specific wrapper is to conform to "the main API", translating into and out of the tool's specific needs. So if we find WhizzySADTool takes an mp3 file and puts output in /tmp/whizzyTool.xml ... we reformulate our wrapper to accept WAV files (by calling on a general conversion file that lives inside toolbox/, since it may and will be used by other tools) and rewrite output in /vagrant/data/whizzy_tool...RTTM (idem for the xml to rttm conversion). This tool-specific wrapper lives in the folder with the tool, not in toolbox/ because those needs are specific to that tool.
• Each tool-specific wrapper will define how to deal with long files and with many files; by and large, we can imagine two possible issues:

1. tool cannot deal with long files, so they need to be cut up into smaller clips, then each clip gets processed, then the individual rttm's get reconcatenated. Note that Eric already wrote something for this. Note also that this is excellent for models that take local decisions -- i.e. all except diarization like diartk, which should try to label each individual speaker over the whole file. Let us make this the default for now.
2. tool has massive model, so it would be better to load this only once and then process all the files. It is difficult to imagine a tool that would benefit from being called for each file separately, so let us make this the default for now.

Don't hesitate if you have ideas to improve the way we're dealing with the files hierarchy. It's now or never !

[riebling]

lots of tools break when they can't find things where they expect. I'm not sure it's fair to require tool creators (some no longer available, or some tools we may not have source code for) to make changes to their code to accomodate new folder names.

[riebling]

Config files normally found in /vagrant have gone missing in the reorganization. More than one tool uses the HTK feature config file (ERROR) [1] in configManager : cFileConfigReader::openInput : cannot open input file '/vagrant/MED_2s_100ms_htk.conf'! Also .theanorc (which is in the reorganized repository) and also a whole folder of configuration files named https://github.com/srvk/DiViMe/tree/master/conf which exists "outside the VM" for a reason: it makes it easier for nonexperts to change a configuration local to their machine, but then tools in the VM read the new configurations to work in a different way. conf is currently mostly for enhancements to the VM (the slurm batch queuing system for example) but I see @MarvinLavechin is using a new subfolder conf/vad. I propose that we preserve conf in the reorganization, and if we want to move forward with changing tool source code to work in new folder layouts, we alter the source of the 2 or more tools (much as this terrifies me) that depend on MED_2s_100ms_htk.conf to get their configs from /vagrant/conf/ instead of just /vagrant, so that at least that ugly MED_2s_100ms_htk.conf filename won't be there making the list of files look ugly :)

[alecristia]

lots of tools break when they can't find things where they expect. I'm not sure it's fair to require tool creators (some no longer available, or some tools we may not have source code for) to make changes to their code to accomodate new folder names.

For the past tools, I propose we either fix it ourselves or remove the tools. We are not committed to preserving all tools that are there. The goal is not to preserve the VM as it is now, but rather to build a sensible, well-organized platform that others can build on and use easily. Perhaps the changes we suggested with Marvin were not ideal for this, so we are very open to discussion of how best to do this.

[alecristia]

Config files normally found in /vagrant have gone missing in the reorganization. More than one tool uses the HTK feature config file (ERROR) [1] in configManager : cFileConfigReader::openInput : cannot open input file '/vagrant/MED_2s_100ms_htk.conf'! Also .theanorc (which is in the reorganized repository) and also a whole folder of configuration files named https://github.com/srvk/DiViMe/tree/master/conf which exists "outside the VM" for a reason: it makes it easier for nonexperts to change a configuration local to their machine, but then tools in the VM read the new configurations to work in a different way. conf is currently mostly for enhancements to the VM (the slurm batch queuing system for example) but I see @MarvinLavechin is using a new subfolder conf/vad. I propose that we preserve conf in the reorganization, and if we want to move forward with changing tool source code to work in new folder layouts, we alter the source of the 2 or more tools (much as this terrifies me) that depend on MED_2s_100ms_htk.conf to get their configs from /vagrant/conf/ instead of just /vagrant, so that at least that ugly MED_2s_100ms_htk.conf filename won't be there making the list of files look ugly :)

I think part of this comments point out a bug I introduced -- sorry about that.

And part is about adding back a folder I did not know about. Of course, we can use it, in which case it would be ideal if there was some documentation explaining what is in there and why, so that in the future people can re-use it.

[riebling]

Super! Moving on... (I've been working from home this morning since waking up a couple hours ago) I've now made enough changes to the new scripts in launcher/ (and possibly utils/) to get the tests to pass. This does not mean that they all work in the desired way yet, simply that they WORK. If it's OK to commit all my changes, you could use this as a starting point for "something that works" then make incremental changes to finish making them work with the new reorganized folder paths.

In particular, things that I changed in order to make the tests all pass, that may need more changes to work to fully work in the reorganized way:

• by hand installed LDC sad tool in the VM so it could be tested
• Put a copy of Opensmile back in vagrant's home folder (tools that are hard coded to find it there) by hand
• Put OpenSAT back in the home folder (it has numerous hard coded dependencies that require it to be there) by hand
• put G/ back in the home folder (tools hard code that location) by hand
• Refactored and fixed bugs in numerous launcher/ and utils/ scripts relating to pathname confusions we've encountered: /vagrant vs. /home/vagrant and DATADIR being absolute path vs. relative (data vs /vagrant/data)
• This includes some cases of scripts that take as an argument a folder name (eg "data") intended to be in /vagrant (/vagrant/data) being actually passed full paths ("/vagrant/data") [ in the eval.sh stack of scripts' use of audio_dir]
• LOTS of fixes where more tool names needed to be changed from e.g: tool_sad or tool_diar to toolSad and toolDiar

Things that still need work:

• eval.sh sub-scripts still use temp_ref and temp_sys rather than temp/temp_ref and temp/temp_sys (there were too many changes for me to make to get it to work the temp/temp_ref way)
• many places in scripts define paths with trailing slashes, that then get concatenated into bigger pathnames that work fine, but if you print them out the paths look like e.g: /vagrant//data/VanDam-Public/BN32//folder-name - not a real bug :)
• G/ folder is back in /home/vagrant, again because tools have hard coded dependencies on it being there
• Changes to Vagrantfile to ensure these tools are back where they were -or- a larger effort of changing the tool sources (something I was hoping we could leave fixed) to work with new locations which involves:
  • putting modified tools in a new branch or release tag or repository
  • maintaining multiple versions of tool source code, if we wish to support people currently using them in current DiViMe deployments
  • possibly updating tools' internal documentation
  • possibly updating system-level documentation

So I'm mostly writing to ask if it's OK to commit my changes to test.sh that "work", and the launcher/ scripts I've modified to work with it, in hopes that development of the reorganized branch can continue with now a way to test tools again, and starting from these "working" ones, then make further refinements to them use new temp folder layout.

[alecristia]

yes, certainly! Please push at your convenience, I'll pull and take the relay.

[macw]

I agree. We want a well-organized system and there is no sense in having extraneous pieces. Sometimes people like to keep things in systems just in case they might be useful, but then those extraneous pieces end up causing organizational drag.

--Brian

On Nov 15, 2018, at 3:05 AM, Alex Cristia notifications@github.com wrote:

lots of tools break when they can't find things where they expect. I'm not sure it's fair to require tool creators (some no longer available, or some tools we may not have source code for) to make changes to their code to accomodate new folder names.

For the past tools, I propose we either fix it ourselves or remove the tools. We are not committed to preserving all tools that are there. The goal is not to preserve the VM as it is now, but rather to build a sensible, well-organized platform that others can build on and use easily. Perhaps the changes we suggested with Marvin were not ideal for this, so we are very open to discussion of how best to do this.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/srvk/DiViMe/issues/64#issuecomment-438952140, or mute the thread https://github.com/notifications/unsubscribe-auth/AFXzuaxzuULsY0YHq_5Kxuqlx2Ujy28Xks5uvSBbgaJpZM4X4RsU.

[alecristia]

Eric and myself discussed having a launcher and utils folders as separate repos versus within the divime folder. We decided for the latter because:

• for someone who wants to install a new tool inside VM, this means forking ONE thing and not 2 or 3,
• for someone reviewing a pull request, this means reviewing changes in one repo, not 2-3
• it'll be easier for tool makers to edit launcher etc. from the host machine rather than only through the VM
• all the launch/util scripts are hidden neatly inside folders so the GitHub page for DiViMe home folder still looks clean enough

The only disadvantage is that this makes the divime repo heavier in size, but this does not seem to be an issue given that launchers and utils are all light files.

We also decide to add repos to PATH, which will allow very clean and easy to read code for calling tools: vagrant ssh -c launchMe.sh data

We also decide that launchers will be by default read-only, to avoid accidental editing

[riebling]

ok! not sure what all check boxes we can check off, but I just committed a Vagrantfile and updated some tools:

• Yunitator and OpenSAT (noisemes) now run "self contained", no dependence on hard coded paths. As a result they live in repos/ now.
• openSMILE installs SMILExtract in /usr/local/bin/ so that's available systemwide
• ~/G/coconut and ~/G/pdnn are baked into Yunitator and OpenSAT now, so - drum roll No tools are installed in the home folder, all are in repos/. (except Anaconda but that's just how Anaconda rolls) If you manually install ldc_sad_hmm from the aclew repo in this VM, it should pass all tests. This does not mean the reorganization is complete, simply that the tests are now usable as tests, and further changes to pathnames, logging, temp files, documentation, etc. can proceed :)

[alecristia]

• [x] straightening out chaos That's one box you should check!

[riebling]

one of the far above checkboxes goes away (compile selectively the docs based on tool failures)

[alecristia]

good point, that actually takes care of 2 problems - because now the online and local docs are the same. I'm adding a line to bootstraph.sh to build the local docs.

Also, I think the last unchecked box, "Inside each tool subfolder (e.g., tools/OpenSAT), there will be a wrapper that specifies the behavior to adopt regarding the files that need to be processed*" is no longer relevant, right? Since you're systematically having all the tools analyze file by file, so by the same token, this routine should break up long files into subfiles?