Reproducible/Cross-Platform Builds

sneakers-the-rat commented 1 year ago

Tracking issue for making reproducible builds and ideally cross platform builds.

Work will be done in https://github.com/Aharoni-Lab/Miniscope-DAQ-QT-Software/tree/feature-repro-build

Overview

Currently the build has unspecified dependencies (hardcoded paths to opencv in particular) and so we would like to standardize the build and deploy process for CI/CD. We would also like to target multiple operating systems (i think all with x64 arch?).

TODO

To do that, I think we need to

Use multiple github actions runner images for windows, linux, mac: https://github.com/actions/runner-images#available-images
Make dockerfiles for each that include proper versions of
- Python
- OpenCV
- Qt5
Modify the qmake project with platform-specific dependencies/configuration options
Configure build action on push to main to make
- versioned
- releases

Need to know:

If we need to make any modifications to the source to accomodate differences in OS support, @phildong mentions differences between directshow and v4l2 in support for setting some camera properties.
- In V4L2 this is dependent on the particular driver used for the miniscopes, so need to figure out which that is.

Prior work

Repos

Cross-platform DAQ software: https://github.com/bothlab/pomidaq
Fork that adds linux support (and does a substantial refactor, we should decouple and do one and then maybe the other if desired): https://github.com/HaaPut/Miniscope-DAQ-QT-Software/tree/master

Issues

Intended to close:

jonnew commented 1 year ago

Hey all, I want to add support for our commutator to some variant of the Miniscope DAQ software. Do you all have a suggestion as to what I should be targeting? e.g. is pomidaq a hard fork/different implementation of this project or is there an intention to merge?

sneakers-the-rat commented 1 year ago

I think @fnsangiul knows more about the status of the commutator stuff. and @daharoni knows more about pomidaq than I do, but as far as I can tell it's a hard fork. If this was my repo I would def be reaching out to them and trying to pull in some of their changes, at least the build part, as the current project file leaves a bit to be desired as far as a stable target goes.

We're moving towards normal-practices CI, repro builds, versioned releases, and testing that would make it possible for you to just PR whatever you're working on into this so we can consolidate labor, but still some friction en route. from my end i'm mostly waiting until some stuff that seems to be up in the air with the other new miniscope projects settles down and we can focus on getting the infra to work, but if we can do that before another hard fork and can work together on this that would be great.

I'm also not sure about the longer term plan for this particular piece of software, in my opinion we should separate out the actual logic of reading/configuring the miniscopes from the GUI so it's possible to build on top of them in a more flexible way. My demo project of that is over here for the SD-card based miniscopes: https://github.com/Aharoni-Lab/miniscope-io That's not intended for realtime acquisition/control, since the SD-card miniscopes can't do that at the moment, but some more minimal SDK like that which doesn't require a whole ass Qt would be great. at this point sort of minimal buy in so i also am not sure if the rest of the lab shares that goal. Again @daharoni knows more about the longer term plan for this software than I do.

sneakers-the-rat commented 1 year ago

Confirmed we are working again on getting the build reproducible, @fnsangiul has started on a build guide and then I'll transcribe that into CI.

We'll also need to upgrade to Qt6 because Qt5 is EoL, and that should also change the difficulty of building/contributing to the software. https://github.com/Aharoni-Lab/Miniscope-DAQ-QT-Software/issues/62

Still waiting for comment from other authors on prior efforts re: the commutator (which i know was being worked on) and the expected life of this software, but we'll at least get it to a point where it can be built.

jonnew commented 1 year ago

Thank you @sneakers-the-rat really appreciate it.

jonnew commented 1 year ago

And, I think its widely used enough that its worth getting CI or reproducible local builds up and running. We have a lot of customers that want to use the commutator with this software.

sneakers-the-rat commented 1 year ago

Worked on this for a few hours today, I think doing the upgrade to Qt6 at the same time is worth it (rather than doing a build -> upgrade to Qt6) because Qt6 is now built with CMake, which simplifies the build process and also we would have to rewrite it in a second step anyway.

https://github.com/Aharoni-Lab/Miniscope-DAQ-QT-Software/commit/2857e67329eea4da5d84b5470040bea736a9b941

will need input from @daharoni on how to go forward because a few things are deprecated in Qt6 that I can't quite rewrite because I don't know how they work in the first place - most notable the TreeViewerJson.qml widget, where the object model vs. view structure has changed dramatically and there aren't good docs on making an upgrade that I can find.

But basically from this, given system-accessible OpenCV, Qt6.5, and numpy, you just

cd source
cmake -B build
cd build
ninja

and there you go.

After I fix the bugs that come from the Qt6 migration then I'll write the CI build, and at that point we should be more capable of accepting PRs/less moving pieces to merge into.

daharoni commented 1 year ago

Great to see this moving forward. It has been a long time since I worked on this code but hopefully I can be of some help.

I'll refresh myself on the structure of this code and we can discuss solutions to remaining challenges later in the week.

hsw28 commented 9 months ago

I am crying from happiness at this

hsw28 commented 9 months ago

fwiw I was doing this in windows because I thought it would be easier than in linux lollololol

Aharoni-Lab / Miniscope-DAQ-QT-Software