Custom Build Implementation for CMake

[HTRamsey]

Custom Builds compatibility needs to be updated for the new CMake build system.

[DonLakeFlyer]

Things that are a pain with respect to current way that custom build and update process works:

• The fact that the custom code and the upstream qgc code are in the same repo causes all sorts of nightmares with respect to bringing upstream qgc up to date in your custom build repro. You spend tons of time resolving merge conflicts and what not on the upstream pull.
• The process of forking the regular qgc repo and then renaming custom_example and go from there means that your starting point for custom files are forever linked back to the original upstream ones. Which in turn means as your custom build gets more complex, you run into tons of merge conflicts when updating to a new upstream qgc.

I think that a mechanism where the custom build source is disconnected from the upstream qgc source would be way better. The process then being:

• Create a brand new empty repo for you custom build
• Copy whatever example custom build stuff you want as a starting point into there
• Somehow reference upstream qgc in your build more like a submodule type thing

This way:

• upstream qgc is disconnected from a git pull standpoint. If you have a clean custom build implementation which has no upstream qgc changes. Then you just update to newest upstream qgc. No merge conflict hell.
• Your custom code is not connected to the initial example code. So again no merge conflict hell as the example custom build goes through modifications.

[mrpollo]

Thanks, @DonLakeFlyer. I have heard from devs maintaining custom forks that things could be better, which is also one of the reasons the project sees few contributions. When someone writes fixes/features downstream, pushing them upstream is hard when the delta is too high, especially when downstream stays the same for a while.

Testing coverage for this effort would likely be required. We need to offer a way for folks to test upstream changes against their custom builds without completely derailing their development efforts.

One way to make this a successful migration would be to have the QGC App itself be a third-party app using the same system we are putting in place. What I mean by QGC App is the app you download from the app stores.

[DonLakeFlyer]

When someone writes fixes/features downstream, pushing them upstream is hard when the delta is too high, especially when downstream stays the same for a while.

Yeah., there is an issue with folks just hacking on mainline QGC code to get their custom build to work. That in turn can lock them away from ever updating from upstream again since it's too painful. It's the quick way to get things done but can lead to pain further down the road. The Issue is that I get little information on what is needed in the custom build architecture to not require or limit upstream changes. Most of my custom build architecture comes from my own needs in my custom build and trying to keep my builds clean from upstream changes. If folks could just bring up problems they run into that would be helpful.

[DonLakeFlyer]

I'll give you an example which I hit in my own custom builds which I plan on fixing soon. I need the ability to add my own custom actions to the Fly View Toolstrip. That is not currently possible since GuidedActionsController is a big monolithic thing which you can just add to. For my custom build than the only thing preventing me from using QGC upstream as is. Right now I've hacked my commands directly into upstream. It's a fixable problem just haven't gotten around to it yet. So I myself am no better!

[Davidsastresas]

I work regularly on custom builds based on the current custom plugin architecture.

For situations like guided actions controller, as it is right now we can override the full guidedactionscontroller.qml in our custom plugin, or any other .qml files for the matter. It is not ideal of course because when rebasing to newer upstream qgc one needs to account for it on these kind of files, in case anything changed upstream in the original guidedactionscontroller.qml.

It is true that guidedactionscontroller.qml is a particular file, because it is very common to modify it. So maybe on this particular one, the issue isn't how custom plugin works but the file itself. Maybe we can somehow refactor that in a way that it depends on core plugin, so we can override it from custom "cleanly". Maybe something similar to how firmware/autopilot plugins work would be good. Like a base guidedactionscontroller where we can extend the functionality inheriting new classes from it. Pretty much the firmware plugin idea.

For the situation of the custom folder being linked forever to the original custom-example one, I personally leave custom-example folder just there, and create a new custom folder and work from there. This way I think you can avoid that merge nightmare in custom-example folder, as custom folder will be just a new folder, no merge conflicts there.

About being the same repo, I don't think that is necessarily a bad thing. In fact I like that. Whenever I work on a custom build, If I find something worth of a PR upstream I just change the remote url and push the changes to make a PR. If custom build used regular QGC as a submodule, this last scenario would be a bit Les convenient I think.

Overall, I am not dissatisfied with how it works now. There is always room for improvement of course but I don't see this situation that bad. I don't believe this is the cause for people not contributing to QGC, or for people not using custom plugin. For somebody that just came to develop stuff in QGC I agree working over custom plugin is an "overhead" at the beginning, they will need to get familiar with it. But in the long term it is much better to make the effort of course, as the time savings when rebasing is huge if you make your custom plugin arquitecture smartly.

Having said that, I am not closed to improving the experience of course.

[DonLakeFlyer]

I personally leave custom-example folder just there, and create a new custom folder and work from there.

Duh! Such a simple solution that complete evaded me :)

With respect to GuidedActionsController and custom plugins in general: My thinking is moving more towards setting up the Qml to make it easier to do resource overrides instead of using custom plugin c++ code to inject ui. With Qml changes such that the override doesn't end up being the entire file.

[DonLakeFlyer]

@Davidsastresas Could you detail a list of things you typically have to change in upstream qgc code? Then I can think through ways to to make this easier, possibly with less upstream changes.

[HTRamsey]

Couple of small comments:

I think the QML stuff does a good job of having smaller independent code structures that are grouped together well, like all your settings pages or toolbar indicators in one folder and each one is just a couple hundred lines. But there are a few places, particularly with cpp code, where the code is very monolithic making it extremely tough to follow upstream changes. Like the Vehicle file being almost 5000 lines is kinda rough. There's also a couple folders with a large collection of only loosely related things.

So that's one reason I was trying to pull out all of the mavlink stuff, make it agnostic to the actual vehicle itself, and place them in well defined smaller modules. The mavlink code usually won't change much so that's a massive portion of code someone wouldn't have to worry about merging changes in. That's why I've been trying to work on cleanup/reorg rather than tacking on more features right now.

With the new qt_add_qml_module, I think it'll be easier to have smaller and better defined qml modules too. Right now you usually just kinda import everything in the Qml files, but maybe someone will never use Video (for example) so it would be nice if that were a separate module where code changes, resources, and files could all easily be dropped and ignored.

Right now it's literally impossible to tell which resource files like pngs/svgs/qmls are actually used and where. So if the modules had their relevant resources in the folder with them that would make it easier to figure out what is being used and by what, and which modules depend on each other. Basically making each module its own independent project. (Some of that is already kinda done with the Firmware/AutoPilot plugins).

Then maybe other projects could reuse modules easier too? I think mavlink projects tend to end up recreating the same boiler plate code too much. Which is probably partly why you don't often see smaller projects succeed long-term, because there's a lot to implement to get basic functionality. Whereas basic libraries like the LibEvents mavlink module are nice so people can kinda reuse and improve a single implementation.

As for contributions, I think some people get hung up on the idea of using Qt maybe? That page on discord "next-gen-gcs" cited replacing QGC with something not using Qt. Maybe having some of the modules being pure std library would make it more obvious that you don't really have to learn anything to use Qt, you just get easier cross platform compatibility with it. You could even run QGC in a browser now. I know Flutter is getting popular but I don't know much about it.

[Davidsastresas]

@Davidsastresas Could you detail a list of things you typically have to change in upstream qgc code? Then I can think through ways to to make this easier, possibly with less upstream changes.

Sorry @DonLakeFlyer, I've been meaning to answer you for several days, but I was really busy with work on the latest few days.

Here is a summary of my insights on custom builds right now:

Not super hard to fix

• QGCPostLinkInstaller.pri: It doesn't support custom app names for android, it is hardcoded to "QGroundControl+bitness" right now. This is an easy fix, and it is on my queue of stuff to push to master.
• qgroundcontrol.pro: For some reason I never managed to do just within custom.pri some details like Windows app icon, windows installer icon, and windows installheader.bmp. For the rest of the platforms I am able to though!

Medium complexity to fix

• LinkManager: It would be cool to be able to override this somewhere. Right now it is hardcoded in Toolbox. Sometimes support for odd devices need to be added, devices that don't have real value for upstream QGC, and sometimes changes on linkmanager need to be done.
• GuidedActionsController: discussed previously on this thread, this is something really prone to be customized and right now the only way is to override the full qml class within custom plugin. I think a quick approach would be to make this a "base" qml file, and include on it some callbacks for possible custom handlers. For instance adding support for executeAction and confirmAction to "probe" for possible custom implementations of unknown actions within that file ( in "default" case on the switch loop ).
• GuidedValueSlider: This is also modified a lot, reused for new guided actions. This one doesn't annoy me too much, but right now the onsuccesfully way to override it is also full qml override in custom plugin.
• Joystick: Would be good more flexibility to add custom button actions. I have in the queue a couple of PR related to this too. We have right now the .json method to add them, but it would be cool to leverage this for custom builds, so we can leave "embedded" those .json files in custom builds, not needing to link the custom .json externally. Similar to that PR I pushed recently for custom camera definition files. This way just with a .json file we could have custom button actions, without needing to add that file externally in "user space".
• AltModeDialog.qml: Would be cool to be able to define from custom plugins if we want to "remove" some particular altitude frames. Some custom builds want to use exclusively terrain alt, some custom builds want to remove completely "calculated above terrain" because it confuse users, etc. We could define the suported altituded frames on CorePlugin so Custom plugin can override it.
• Appsettings: Would be very nice to have here a chance to create custom folders. Problem is coreplugin is not "alive" yet when appsettings constructor is called, settings manager is the first to be instantiated in toolbox, so this one is tricky, and I usually just modify AppSettings to do this job. Not a big deal though.
• Custom settings: I know there is a way right now, but it is odd and not super intuitive. It would be great if we can make something in SettingsManager to add them directly there ( so from qml they can be fetched from settingsManager, and not from some odd place ). I am thinking in something like we do in toolbox for custom coreplugins:

  #if defined (QGC_CUSTOM_BUILD)
  //-- Create custom plugin (Static)
  _corePlugin = (QGCCorePlugin*) new CUSTOMCLASS(app, this);
  if(_corePlugin) {
      return;
  }
  #endif

  Or by being able to create a new class inheriting from settingsManager somewhere.

Much more involved to fix:

Here I basically want to mention custom missions. As it is right now, it is very complicated to follow a custom approach for anything rsuccesfulelated to mission, and even though not all custom builds are interested on this, it is not uncommon to need customization for missions. Some random examples:

• Lidar surveys, they don't need camera waypoints at all, and they consume a lot of memory on the autopilot, and on some long endurance fixed wings this is a problem, so only way is to make this custom and remove the insertion of camera waypoints on the standard code
• Custom surveys: Survey code can be reused for other types of surveys. For example spraying missions ( similar in concept to photo surveys, but they require different waypoint types to trigger sprayers ).

• Custom complex items: Sometimes custom behaviour on complex items is needed. For instance, we had a request recently for solar inspection surveys. These surveys need to have drone camera at the same angle as the solar panels, so you have to manually account for the offset of the actual solar panels placement, and the real location of the panels. In this case we have a custom complex item where you select the solar panel area and introduce the panel orientation, and the offset is automatically generated.

  There are several examples like this. I am aware this one is really much more involved than the rest of them, so it might not be something we can attenuate soon.

  Final thoughts

  Even considering all the above, I think the current infrastructure QGC has for custom build is awesome. Custom QGCCorePlugin custom firmwareFactories and autopilotFactories and being able to override .pri, resources and qml actually covers most of the work we usually do in custom builds.

There is always stuff to improve, but I am personally very satisfied with it, and even right now it allows for a nice and comfortable management for custom builds, even if every now and then some "non custom" code needs to be modified.

So I am just reporting this because you asked, but it really isn't a big deal.

I have some pending PRs to do about some of the points, and together with what you said:

With Qml changes such that the override doesn't end up being the entire file.

I think we shouldn't obsess to much about this matter. I think it is good to keep making this custom architecture better overtime, but it is reasonably nice already, I wanted to express my thought on this clearly. I think it is more important all the work that is being done right now in master.

Why do we really want to improve this?

In the end of the day, what do we want to solve making even better custom plugins? To get more contributions? I don't really think it will improve this front.

I think the only way to improve the contributions front drastically is by improving QGroundControl at a rate that is "uncomfortable" for custom builds to follow up. This will force companies to push more to open source on order to "release" maintenance pressure. After many time thinking about how Ardupilot is so successful at this, I really think this is the answer. Because the workforce is out there, there are plenty of custom builds out there, there are a lot of active QgroundControl devs out there.

Of course, in order to get to that situation, we basically need resources in QGroundControl, starting with some fixed positions that service regularly issues, organize pull requests, etc. I think if the current devs could forget at all about maintainance, and only work on features, we would see a drastic improvement in "QGC evolution throughput", forcing companies with custom builds to contribute more to maintain less. I hope that makes sense.

Right now we are very few on the dev team, we do our best to spare some free time and contribute to the project, and spare time for maintainance, but I think for most of us we really can not spare more than a few hours per week.

To make it more obvious, this is another call to Dronecode to organize this better. I think the project deserves it, and I think it has more than enough user base for something nice to be arranged around it. Again Ardupilot project manages to do this just fine, and they have arguably less resources than Dronecode.

[Davidsastresas]

Couple of small comments:

I think the QML stuff does a good job of having smaller independent code structures that are grouped together well, like all your settings pages or toolbar indicators in one folder and each one ....

@HTRamsey You mentioned some really good points. And I am personally very happy to see you around spending all this time lately on QGC. Thank you very much.

I wanted to comment on some points:

About mavlink

I think the current paradigm we have in vehicle right now is good. Maybe we could "transfer" a lot of the mavlink logic to the subsequent blocks ( FirmwarePlugin and autopilotPugin ), so vehicle ends up being purely a "front end" to interface with the firmwarePlugin and autopilotPlugin backends.

About libevents

I am not familiarized with it. I took a quick look and it is pretty cool. But I don't quite understand how we could learn from it other than what I exposed in my point above. QGroundControl is a GCS, so it will always have the "base" of a mavlink GCS. My understanding is libevents is much more simple and general purpose. Besides what I mentioned on the previous point, how do you think we could really make more modular custom mavlink work on QGroundControl? Considering we really should leave there the "GCS mavlink base" for other devs to build upon it?

About next gen GCS thread

I kind of touch this on my previous answer, but it is too long ( sorry about it ), so I will write a quick thought about it here. I think that thread wasn't realistic at all, and I think the problem with QGC contributions isn't few people familiar with Qt. I don't think Qt is much harder to learn than other UI frameworks. But a GCS software as sophisticated as QGC is complex, and it will be regardless of the framework used.

For me, other frameworks are out of the question. I think the balance Qt has to offer for ease of use - robustness - reliability - maturity - flexibility - cross platformness is hard to beat nowadays.

There are a lot of new frameworks, some of them are really sexy, and I agree some of them might have benefits for new developers. But even if we did that work and make QgroundControl "platform agnostic" ( porting stuff to std libraries and such ), would we really get more contributions? I really don't think so...

I think the problem with contributions is to innovate the project at a rate fast enough for companies to think twice before keeping features closed for them. If at some point in history QGroundControl is in a state where bugs are solved within days, where issues are handled properly, where PRs get little time to service, etc I think this front will improve drastically.

But for this to happen we need resources, and well, I would be repeating what I said on my previous answer on "Why do we really want to improve this?" point

[HTRamsey]

I can certainly help with innovating the project as I've got a ton of big things in the pipeline to add. I just don't like doing UI work which is why I'm glad @DonLakeFlyer is so good at it.

[zerocool-z1]

hello, i have quit good experience in developing custom bootloaders, tamper proofing perameter and devloping custom hardware using mavlink and python for drones. I new to QGC development i have build the custom apps using qgc in past (normal ones, mostly for a changed UI).

is there anyone who can teach me about the qgc architecture and developing fully flashed custom app with my own custo UI and customised function?

[garrik]

Hi, I started exploring QGroundControl sources some days ago and begun with building a custom version of QGroundControl. I wrote the cmake implementation for the custom build as close as possibile to the previous one. You can look at it and if it is good enough, I will be glad to contribute with fix or improvents and eventually a pull request. Btw your work is amazing. Regards

[HTRamsey]

@garrik I checked it out and it seems pretty solid. If you want you can put up a PR and we'll get @DonLakeFlyer to review it when he is available.