Code review

[mertcelebi]

Hi @mezarque, just went through the repo, here are some rough thoughts:

• Do you want to include all the mkdocs stuff? It may be worthwhile to move to the biofile specific repo and remove them from here, but up to you. It may be reasonable to delete all the tutorial folders to clean things up.
• It'd make sense to remove the debug folders.
• I'd move the README under notebooks to the main README since it has a lot of useful information and gives a clear sense of the most important parts of the repo.
• Similarly, you can move a lot of the information on the Setup.md to the main README (talking about what type of instance you ran things on and how you configured it etc). If you need inspiration for some of the level of detail around setting up Snakemake, the actin prediction repo has a great README.
• For the conda env, do you have a list of all the tools you used? Two approaches here: we can try to create a clean conda env or try to be like "here's the full conda env we have, but it'll only work on X type of OS on Y type of machine". Up to you, but I prefer (1).
• Main README thoughts (in addition to the previous consolidation thoughts):
  • I'd update it to link to the pub draft (or at least have a placeholder).
  • I'd add a note about species you study and all the relevant datasets (with citations) I know these are already documented in the code, but good to be forthcoming.
  • If we go with the mkdocs stuff, a link to the tutorial website.
  • If this project is iced, potential future directions we decided not to pursue could be included. Or any requests for feedback (maybe on biofile?)
• From a folder organization perspective, here are some thoughts.
  • First a caveat: this is purely subjective, feel free to ignore it.
  • Instead of organizing things by species, would it make sense to organize by stage (download, embedding, analysis etc) and within those name things as you already did (based on species etc)? So the top level folders would 1_download/ or smth like that and within those you'd have download notebooks for each species etc?
  • It may make sense to add note about the annotation download notebooks. Those are included in certain folders but are not addressed in the README(s).

[mezarque]

Progress update 1 on this issue

Addressing above comments

• I went back and forth on the BioFile and tutorial stuff – I think it's better to keep it for now. The newer version of biofile will be substantially different from the version in this repo, and without all of those files and documentation in place, it's probably very difficult to understand what the notebooks are doing. There are a lot of operations that are handled internally by specific BIoFile classes which would be difficult to interpret without the full documentation of parameters in the mkdocs files. I think in the repo's current state, if someone spun up an EC2 instance with permission for Arcadia's buckets in AWS, the current repo should have all the code needed to fully run every notebook.
• debug folders nuked!
• Reorganized the folders to be stepwise (1, 2, 3, 4)

Miscellaneous updates

• Did a bunch of tidying and added more comments to the notebooks to explain in broad terms what they're doing
• Added a bunch of supplementary notebooks for generating some of the pub figs

Next steps

• README updating to reflect the current repo structure
• Merging info from all of the scattered markdown files into the main README
• I'm going to try to build a conda environment from scratch and see if I can get to the minimum set of installs needed to run the repo... wish me luck!
• Continued work on the text of the pub
• Making a Binder or Google CoLab that downloads files into an exploratory environment for the cell spaces; it turns out the compressed .h5ad files are only 10s of Mbs so they should be easily downloadable from Zenodo and visualized using a lightweight instance; generating the compression and doing large data transformations is what requires large amounts of RAM, but for the purposes of visualization/ exploration the compressed files should work

[mertcelebi]

This is awesome, thanks @mezarque. Binder is great if it's multiple notebooks. Though I've run into some rate-limiting issues, so FYI. Colab would be fine if, it's a single notebook with more compute available.

[mezarque]

Progress update 2

New stuff addressing original comments

• I updated the README to reflect the improved repo structure and to collate info from a variety of sources. I've left the original README files where they are for now.
• I added some documentation to the README about what kind of general hardware requirements are needed to run the code in the repo.
• I compartmentalized the hard-coding of the AWS S3 Bucket we'd been using to a global variable in install_locs.py called S3_BUCKET_ADDRESS
  • In theory, users could now modify this string to point to some AWS S3 bucket that they have write permissions for, and the rest of the code would work to handle processing, upload to their bucket, and download from their bucket across notebooks.
  • I also commented to provide a bit more clarity to install_locs.py and indicated in the README that users would need to modify these paths to reflect their local installation environment.
  • This gets us closer to this repo being reproducible in arbitrary compute situations, although fully running our code still requires AWS S3 configuration and a bucket to push and pull from.
• I made a new conda environment detailed in glial_origins_tidy.yml which seems to be working.
  • I built it by looking for all of the "high-level" dependencies I could find in the original .yml and rerunning all of the notebooks, adding things I forgot to include as needed. I didn't encounter any totally broken sections, although some software might not have been touched by the rerun I performed, so it's not 100% guaranteed to work.
  • The main caveat is that the plots do not come out exactly the same (there's some stochasticity in some of these dimensionality reduction algorithms which seems to vary depending on the exact combination of numpy/ pandas/ scanpy versions). I pinned the closest versions for things I think might affect the stochasticity but haven't gotten perfect reproduction yet. That being said, the plots do work, so I'm not going to keep fussing with it.
  • I think that the glial_origins_tidy.yml conda environment should now have everything needed to broadly reproduce the analysis. Exact reproduction should be possible using the full glial_origins.yml file, which I'm going to leave in there for now.

Miscellaneous new stuff

• There are a number of new plots I made for the purposes of the pub. Most start with a letter prefix a_ etc. and all should be described in the "Pipeline Organization" section of the main README.md
  • Some of these plots are interactive HTML files generated using Plotly. For example this scatter plot. You'll need to download and open in your browser since it's too big for GitHub to render.
• I cleaned up a bunch of trailing code and empty cells in all the notebooks that aren't relevant to the pub analysis.
• I've rerun the notebooks again after restructuring the repo to make sure things aren't broken (they're not, surprisingly!)

Next steps

• I'm going to start tinkering with building a tiny Binder environment for people to be able to explore this data.
  • I'm uploading files to Zenodo and will have a postBuild set of calls to wget a la this tutorial to download the .h5ad compressed data files, which should be pretty small (~250Mb) and loadable.
  • I'll make a env.yml file in the top of the directory to build a lightweight conda environment for the Binder.
  • I'll probably make a new directory under notebooks to store the Binder-specific code. It likely won't use biofile_handling
• Before I get on Binder, I'll need to make this repo public.
  • I'm going to open a PR and once it's resolved, perhaps we can make the repo public.
  • After resolving the PR, I'll make a new branch just for Binder development
• The text of the pub is getting closer to the finish line. I'd love your feedback on the text to get it aligned with the recent Software team pub! I'll Slack you a link.