Thoughts on creating an automatic updater for osara

[MohomedRashad]

Hi folks, I've been using Osara for two years and was wondering if it would be feasible to implement an automatic update function that could be triggered by a keystroke instead of manually installing updates, as there are regular updates that are issued on a weekly basis in my experience. I'm currently pursuing a software engineering degree, during which time I've learned a few programming languages and am somewhat comfortable with C++. As a result, I've decided to contribute to the project by developing an auto update function, and I'll need some guidance to get started on my first contribution. Because Osara is available for both Mac and Windows, I was thinking about how to make a feature that works on both platforms. Based on my limited study, it appears that the snapshots are generated automatically, and there is currently no method to retrieve the most recent snapshot because the URL changes very frequently. Because C++ is a low-level language, I'm wondering if it's possible to create an update feature, or if it'd be better to convert to a more user-friendly language like Python. Please accept my deepest apologies if I am too inexperienced for this endeavor. I'd want to contribute to the project because it's been a fantastic experience using this extension. If you've considered integrating an update feature and have some suggestions for how I might go about it, any advice and suggestions would be extremely appreciated. I'd want to state that I'm open to any and all suggestions, and that I'm willing to learn any new technology that may benefit the feature. Thank you so much for the fantastic work you've been doing, and I'm looking forward to hearing back from you.

[ScottChesworth]

I'm not far enough along with development yet here to offer practical advice beyond just getting the dependencies set up and the real basics of using Git, but as you know, an update feature is requested often, and greater adoption of the latest snapshot would definitely make my life easier across all of the support channels. For now, I'll just vouch that @MohomedRashad is a dedicated dude who learned the recording ropes fast, has stuck around to help in the WhatsApp group and I'm hoping that one of our more experienced contributors can find time to help bring him onboard. A few thoughts on the goal you're aiming for:

1. Start simple. A report that a new release is available with a button to visit the snapshot page would be a huge step in the right direction. If you can get that working reliably, then we'd already be in line with the functionality that SWS provides. Fancier stuff can come later as your experience grows.
2. Sure, the URL changes often if you consider it as a whole, but I think there are enough parts of it that stay consistent that it can still be useful in some way. For example, the numbers after "pre-". Maybe you could compare that part of the URL against the output of our "OSARA: About" action.
3. Just in case you're new to open source, try to resist the temptation to sit on code until you've gotten it perfect. One of the coolest aspects of developing here on GitHub is that it's easy to get input from devs who are more familiar with the project, so don't be afraid to submit draft pull requests for review. Realistically we don't have the resources to handhold newcomers, but everyone who contributes regularly here appreciates a self-starter.

[ScottChesworth]

If you haven't already, take a read through the comments of #372. Not exactly the same topic, but some of the ideas and pitfalls raised there will likely be relevant.

[jcsteh]

Hi. Thanks for the interest in contributing.

It should be fairly straightforward for me to implement a continuous integration change which dumps the version number, download URLs, etc. in a simple metadata file with a permanent URL. That mitigates the problem of needing to deal with changing URLs. That's the easiest part of this whole thing.

The challenge is checking that URL from C++. Moving to a language like Python just isn't an option here. We don't want users to have to install Python, nor do we want to bundle megabytes of Python dependencies just for an updater.

Doing this in C++ in a way which works on both Windows and Mac is hard, though. WDL (the library written by Cockos that REAPER, OSARA and SWS all use) does have some low level support for http, but it's really horrible to deal with. Still, I don't know of another cross-platform way to do this which doesn't involve pulling in a tonne of other dependencies. So, looking at how SWS does this might be your best bet.

I'm happy to consider adding dependencies for this if they're relatively small, so you could also try looking for a small, simple cross-platform C++ http library if such a thing exists.

[jcsteh]

I still wonder whether it'd be better to somehow use ReaPack for updates if there's some way we can prompt to install the new key map within OSARA, so that might be another thing to look at. It seems wasteful to invest in an updater when ReaPack already exists and is used by just about every other extension.

[ScottChesworth]

The only advantage I can see to OSARA rolling our own updater is adoption. Not everyone has ReaPack, and it has a perception of being more for the power users. I'll take whatever I can get though. :)

[Flameborn]

The only advantage I can see to OSARA rolling our own updater is adoption. Not everyone has ReaPack, and it has a perception of being more for the power users. I'll take whatever I can get though. :)

I agree with @jcsteh, reinventing the wheel is quite unnecessary, when ReaPack performs the same functionality admirably.

The adoption of ReaPack, I imagine, would be optional, and as harsh as it may sound, we are not here to change perceptions. For a lot of users, this feature would be a huge timesaver, but if someone prefers to do manual installs, the current method would remain available.

[MohomedRashad]

Thank you for getting back to me, Scott and James. It is really appreciated! I went over the previous issue referenced by Scott and found a lot of good information and solutions. So, as Jamy suggested, I'm going to look into the SWs updater because I haven't come across any solution, and I completely understand why using another programming language would raise the extension's complexity. Now all I have to do is understand how to implement the Bair minimum in C++, and I'll come back to you as soon as feasible. Thank you for your unwavering support.

[ScottChesworth]

reinventing the wheel is quite unnecessary, when ReaPack performs the same functionality admirably.

It doesn't though. See prior sticking points around key map management that we don't have solutions on the table for yet, one such discussion is linked above.

we are not here to change perceptions

Agreed, which is exactly my point. Distributing updates through ReaPack, while welcomed by many, still won't benefit all users, because we can't change the perception around it. Some sort of notification system baked in to OSARA and enabled by default will reach everyone. How they choose to act on it is in their hands of course, but if more consistent adoption is the goal here, then ReaPack doesn't solve it as well as something built-in would.

[MohomedRashad]

I was having a quick question. Is there a way to get the current OSARA version so that I can compare with the new releases? Btw, sorry if the question was too basic.

[jcsteh]

The other thing that just occurred to me is security. If we direct link to a download, we'll need to get that info via https and validate the server's certificate using the system's certificate trust store. Otherwise, a man-in-the-middle attack could be used to make the user download an arbitrary executable. I don't think WDL handles https certificate verification (if it handles https at all). Note that validating the download URL isn't an option because AppVeyor URLs are arbitrary; there's no way to know it's an official OSARA snapshot.

Linking to the snapshots page would deal with this, since the worst an MITM attack could do is notify the user about an update when there isn't one. However, that's far less useful and not a future proof solution if we do ever want to direct link to the download.

[jcsteh]

Not at my desk at the moment, so can't check the code, but I think the OSARA version is defined as the OSARA_VERSION macro. Check cmdAbout in reaper_osara.cpp.

[jcsteh]

What does REAPER do when you accept a new update notification? Does it just link to the download page or directly to the executable?

[ScottChesworth]

It just links to the download page

[jcsteh]

I guess it's reasonable for OSARA to link to the snapshots page after all, then. That avoids the MITM + https challenge at least, though I'll have to configure the server to allow plain http for the update metadata. (Right now, it forces https for everything because security.)

[MohomedRashad]

@jcsteh Any help from your side is greatly appreciated about the URL handling part. A quick method to check whether new version is available for OSARA.

[jcsteh]

The server side solution really depends somewhat on what is reasonable to implement in C++.

1. If the solution you find does reasonably implement https + certificate verification, great: we can have the metadata include the download URL.
2. If it implements https without certificate verification, that at least means the traffic is encrypted, but we can't link to the download. In that case, there's no point in including it in the metadata; we'll just link to the snapshots page. That's suboptimal for security, but since we're just opening the snapshots page in the user's default browser (which will be https), it shouldn't be a real problem.
3. If it does neither of these things, we'll need to go with plain http, which is even worse for security but not a big deal for the same reason.

I can implement any of these on the server side, but I don't want to implement anything until we work out what is feasible on the C++ side.

[jcsteh]

I just found cpp-httplib, which might be worth looking at if WDL proves not to work out. You'll still have to deal with running this on a background thread yourself, though, which will be ugly to do cross-platform.

[MohomedRashad]

@jcsteh Alright. Thank you for the additional information. There appear to be fewer open-source networking libraries for C++ than I had anticipated. I'm undecided on which path to take. And I looked into the WDL library, and it appears to be a lot more involved than I imagined, as it is a versatile library that, as far as I can tell, is largely utilized in audio-related applications which is a little more complicated as you said before. I've found a library called "libevent": https://libevent.org/ which is an event notification library, and it is cross platform and has a minimum boiler plate code to get started. This is a very comprehensive documentation for the library if you may need it: http://www.wangafu.net/~nickm/libevent-book/

It would be fantastic if you could recommend a library where I could continue my development. Because I'm at a loss for what to do. Then I'll be able to get right to work. For the time being, my plan is to download the html contents of the snapshot page to the memory, and compare the current and new versions, as @ScottChesworth mentioned in an earlier comment. For the time being, that is what I can see as a starting point. Please let me know if you have any other suggestions for a better way to approach this implementation. For the time being, I'm not intending on downloading the OSARA binary files for the mac and the windows, but we can implement that feature if it's necessary in a future release rather than compairing the versions. BTW, it's not an issue if you want me to proceed with the WDL library. I'm pritty sure I can manage it. Thanks in advanced.

[jcsteh]

I think you'll find libevent is much heavier than what we need here. Also, it requires that you integrate it into the event loop of the application, which I don't think is going to be possible within REAPER, since REAPER runs the event loop. (I'm actually not sure it's possible in any Win32 app without running a separate thread for libevent.)

The WDL http client header file is in WDL/jnetlib/httpget.h.

[MohomedRashad]

@jcsteh Alright then, Do you wont me to proseed with WDL? Just for the clarification.

[jcsteh]

Looking at jnetlib/httpget, it looks like you don't need to worry about threading there; you can poll with a timer or the like. It might be simpler to use that than I thought. Curiously, SWS does create a background thread for it though; I don't know why. As I suspected, it doesn't support https.

[jcsteh]

I can't give you a definite recommendation on how to proceed. I just don't know what kind of hurdles you're going to hit. Having done a tiny bit of digging now, I'd probably suggest starting with WDL, but you may do a bunch of work and realise it isn't going to work; I can't guarantee that isn't going to happen. If you do find a simpler http library and/or you decide to implement it as a background thread, that's fine too, as long as that library isn't too big and you can get it to compile.

[MohomedRashad]

@jcsteh Cool! Will start from WDL and see how it goes.

[MohomedRashad]

Got a quick question. Since I don't own a MAC system, what is the best way to test Osara in mac? Any thoughts? Are there any ways to emulate the MAc OS in a vm that I can use in my win10 system?

[jcsteh]

No. This is one of several reasons I hate supporting Mac. However, if you submit a draft pull request, AppVeyor will automatically create public builds for you, so you can at least verify that it builds and also have Mac users test for you.

[MohomedRashad]

@jcsteh Got a little question for you. How does osara creates native actions in reaper? For instance, the action to invoke the configuration dialog. Sorry if this is a basic question. Is there any Osara documentation intended at developers that explains how the extension works or something similar? Because I'm new to the codebase, it appears that understanding the code base is necessary to get started. My current plan, is to create an action which checks and reports about any new updates for Osara from the snapshots page without any GUI dialog boxes. Just to be sure.

[ScottChesworth]

Since I don't own a MAC system, what is the best way to test Osara in mac?

Lately I've been running Mac builds by Jenny K from the WhatsApp group. She's keen to help, responds fast and is detail oriented. Also just got my Mac back up and running last night, happy to check things out for ya here.

[MohomedRashad]

@ScottChesworth Ya, that would be awesome! Will get back to you once I'm with the initial version to test. Thanks for your help.

[MohomedRashad]

@ScottChesworth The mac builds you mentioned, are those virtual machines?

[jcsteh]

@jcsteh Got a little question for you. How does osara creates native actions in reaper? For instance, the action to invoke the configuration dialog. ... Is there any Osara documentation intended at developers that explains how the extension works or something similar?

This happens in the same way it does for any REAPER extension. That's documented in the reaper_plugin.h file. However, OSARA has some wrappers around that to make it more efficient given the number of actions it registers. See the big table under this comment in src/reaper_osara.cpp:

// Our own commands.

There isn't any documentation specific to OSARA development, but actions are fairly straightforward. As with learning any new code base, my advice is to dive in and follow the code for something similar to what you're trying to do. That is, don't try to learn the entire code base, but rather, be razor focused on following how a specific task is done.

You gave the example of the configuration dialog, which is not a bad example, though I might suggest starting with the About command instead; configuration is complicated enough that it's split across several functions. If you search for "OSARA: About" in reaper_osara.cpp, you'll see that references the function cmdAbout. You can then look at cmdAbout to see the signature for that function, etc.

[jcsteh]

The automated Mac builds are generated by AppVeyor, which is a continuous integration service that uses virtual machines and integrates with GitHub. You can't really interact with those virtual machines, though.

[ScottChesworth]

@ScottChesworth The mac builds you mentioned, are those virtual machines?

No, just a snapshot of OSARA that can be installed on Mac. We've got integration with a service called AppVeyor that builds snapshots for Windows and Mac whenever you submit a pull request to the OSARA repository. You can use scons to check whether the code you're working on builds locally before submitting your additions as a pull request, that will spit out an OSARA snapshot that you can install and test on your machine, you'll find an executable file has been made at osara\installer\osara_unknown.exe once scons finishes building. I'd recommend having a portable version of Reaper that you use for testing there, just in case. I'm still pretty much a newb when it comes to C++ here, but I've made a few contributions lately. Happy to get together and go over how the GitHub submission process works if that would be helpful.

[RDMurray]

@MohomedRashad Check out this list of C++ libraries. There are several http libraries under the communications heading. IMHO https should always be used, even if it means adding an extra dependency.

From a quick look at the headers, swell does appear to support threading, however if you use an asynchronous library it shouldn't be necessary.

It is possible to run macOS in a virtual machine for testing. I am using VMware workstation. It isn't at all easy to setup, but there are several guides on the web.

[MohomedRashad]

@jcsteh Greate! Thanks for the clarifications.

[MohomedRashad]

@RDMurray Thanks. Currently I'm working on the WDL library as per the reccommendation of @jcsteh , as it has already been added to the Osara dependencies. But will look into it if needed.

[jcsteh]

IMHO https should always be used, even if it means adding an extra dependency.

In theory, I want to agree, but that probably also means bundling openssl or the like if we want to be cross-platform and potentially our own copy of the certificate trust store, unless you can find a cross-platform library which uses both the system crypto library and the system trust store. Even Python bundles its own copy of openssl, and although it uses the system trust store, its code for that is complex and incomplete. Trying to support that in OSARA is going to get very, very painful. We'll probably end up with dependencies which are at least 10 times the size and complexity of OSARA itself, not to mention the pain of integrating them into the build system.

From a quick look at the headers, swell does appear to support threading, however if you use an asynchronous library it shouldn't be necessary.

I doubt there's a true async library that works properly cross-platform unless you use that library's own event loop. We can't do that because REAPER owns the event loop. Also, most cross-platform libraries use select/poll for async, and on Windows, you can't integrate with the event loop that way. You have to use overlapped IO and we might not even be able to do that, depending on whether REAPER's event loop uses an alertable message loop or not. (Many Windows message loop implementations are not alertable; you have to explicitly implement that. Even toolkits like wxWidgets do not use an alertable message loop.)

The approach WDL jnetlib uses to be sorta async is that you poll every now and then on a timer to see if there's new data. That's kinda ugly, but it will probably work. I doubt other libraries will provide that kind of functionality because it's not truly async, but perhaps some do.

[MohomedRashad]

@ScottChesworth That would be awesome. I managed to build the Osara and ran in my reaper to check whether everything is ok, and it was good for me. Did some modifications here and there to understand how it works such as changing the output messages. Any help would be appreciated man! Will contact you when necessary.

[RDMurray]

@jcsteh That really does sound like a huge amount of work and code bloat for something that doesn't add anything to the core functionality of OSARA. Like you say, an http request to check the existence of a new version is probably the best option. I didn't realise how complicated the event loop integration and certificate store stuff was.

After all, Reaper and SWS updates are handled by opening the download page in a browser, so it makes sense for OSARA to do the same.

[jcsteh]

I have some working prototype code for this locally. It still needs quite a lot of work before it's remotely usable, though.