UAV Toolbox 2020 Redesign

[burritobrittany]

Esteemed Colleagues,

Over the past few days, the two of us (Brittany and Rob) have been discussing improvements in CIRN toolboxes (UAVProcessing and others) to remove incompatibilities and improve organization. The following are the main results of our discussions and suggestions of how to proceed:

1. • We need to finally settle on a set of standard structures to describe all of the important metadata such as cameras, stations, geometries, etc.
2. • These structures should be the basis of our common CIRN code on GitHub. For example, a camera should always be described with all of the standard fields so that new codes can be compatible.
3. • One source of incompatibilities is the differences between describing fixed camera and moving camera systems, for example between a traditional Argus Station whose location is fixed and a UAV whose location and pointing angles vary from frame to frame.
4. • There are also incompatibilities between lens calibration descriptors in the lcp structure and the traditional lens calibration structures.
5. • Our goal is to develop a suite of toolboxes that act as building blocks with logical structure and without redundancy, such that users will be encouraged to code using standard routines and structures.

The following are our recommendations:

1. • [ ] - The key to modular coding is identification of a standard set of Argus structures, for example describing all the expected attributes of cameras, stations, geometries, etc.
2. • [ ] - These structures should be captured in a CIRN structures document and should be readily available using the DBCreateEmptyStruct(structType) command so that the correct structures are easily available for programming.
3. • [ ] - The current description of these structures, contained in the attached document argusDBManualV0.8.pdf should be parsed into a document argusStructuresV1.0.pdf that separates the structures definitions from the database manual (document argusDBManual.pdf) so that users who are not interested in using the database will still use the standard structures.
4. • [ ] - The structures defining cameras and geometries should be aligned to handle UAV data wherein extrinsic calibration data (camera location and viewing angles) can be either fixed or changing (UAV).
5. • [ ] - We need to make coherent the two ways of framing the photogrammetry problem, the older m-vector (or equivalent projection metrix, P) and the version that isolates beta (6 component vector of camera extrinsics) used to build P.
6. • [ ] - given decisions on these issues, we need to re-organize the CIRN toolboxes away from a single UAV toolbox and toward a set of logical toolboxes that build upon each other (see draft suggested structure toolboxOrg20190419_JB.docx, below).
7. • [ ] - UAV software, boot camp demos, and presentations should be edited to be compatible with all of this.

We would like feedback. Then we recommend having a CIRN webinar to summarize these issues and recommendations such that we can make final decisions and action plans. I will be posting this on the CIRN GitHub.

Brittany and Rob

[mpalmsten]

Hi @burritobrittany and @RobHolman,

Thank you for putting so much time into thinking about the structure of the CIRN toolboxes. I fully agree with your assessment of the problem and many aspects of your solution.

The most difficult aspects of the book keeping problems that you point out are:

• The structures defining cameras and geometries should be aligned to handle UAV data wherein extrinsic calibration data (camera location and viewing angles) can be either fixed or changing (UAV).

• We need to make coherent the two ways of framing the photogrammetry problem, the older m-vector (or equivalent projection metrix, P) and the version that isolates beta (6 component vector of camera extrinsics) used to build P.

This topic will work great as a webinar. Let's plan a webinar for early January on the topic.

My other main concern is how do we actually get the work done. As follow up to the webinar, I’d like to create a series of bite-sized tasks that individual developers can choose to do or be assigned.

Regarding the demos in the toolbox org document, I want to point everyone to the Support-Routines branch mconlin_PracticumStarterGuide. @conlin-matt put quite a bit of effort to make the demos more intuitive to new users based on his experience at the 2018 boot camp. We should leverage his contribution as we reorganize the demos.

[mitchharley]

Dear all,

Thanks everyone for putting the time in to this discussion as I think it is very valuable to agree as a group on a satisfactory way forward. I am particularly interested in the thoughts about making the Argus DB adaptable to a generic range of image applications beyond the traditional fixed camera stations so that we can expand the community into new applications we might not even have thought of yet. I would love to make any image acquisition system be able to plug into the Argus/CIRN codes as a means of turbo-charging your optical sensor - whether it is fixed Argus station, UAV, smartphones (i.e. CoastSnap), low-cost trailcams etc. et.

Unfortunately I am unable to attend the webinar due to the timezone and being at a conference. To contribute to the discussion though, can we possibly think about arranging the DB in a way that links intrinsic and extrinsic parameters to each individual image or video frame (perhaps moving away from the timeIn/timeOut approach which is more oriented towards stationary, long-term deployments) ? This is how it is done in CoastSnap, since the intrinsic and extrinsic parameters effectively change with each new community contribution. As a side, CoastSnap also solves for these using the GCPS, rather than relying on the camera calibration toolbox (although it doesnt correct for distortion as smartphones already do this in a semi-decent way internally).

Also, I know this is a challenging topic (and probably deserves a whole series of webinar discussions), but at WRL we are finding the Matlab environment quite limiting to make use of the latest developments coming out of computer vision (e.g. OpenCV) and machine learning/AI (e.g. scikit-learn, tensorflow). Our research and clients are increasingly wanting new products like real-time beach usage tracking or ML classification problems, for example:

https://twitter.com/Drummondchris1/status/1185343845803819008

We are therefore looking at moving away towards a Python-based environment (maybe it should be called pArgus?).

If we head down this step, I am obviously keen to make these developments as compatible with the Argus/CIRN DB as possible and provide other CIRN users with a Python-based option. Getting geeky however, there is a bit of an issue with the current MySQL DB in that it stores the data (e.g. betas) in blobs, whereas Python prefers more-standard xml or preferably JSON storage types. I think I recall John mentioning to me about moving away from blobs, but not sure where this is at?

Cheers, Mitch

[mpalmsten]

Hi @mitchharley ,

Thanks for your thoughtful comments regarding your broad vision for the future of Argus/CIRN. I strongly support the comments that you've made. I think that the change to Python is a useful and positive way forward (and is already underway by several CIRN developers). The capabilities and investment in Python is immense given its wide use. Creating a fully open source version of the CIRN repository as a python package is a worthy goal. I'm also very interested in keeping metadata with each image where appropriate.

My main concern is that we have so much work to do with reorganization and a possible change to Python. How can we ever accomplish everything that we aim to do?

Thanks! Meg