SexySolver Astrometry.net and Sextractor based Internal QT Library that works on Windows, Mac, and Linux

[rlancaste]

Dear Dustin Lang and other contributors to Astrometry.net,

I am sorry this might not be the best place to post this, but I wasn't sure where would be best to do it. It is not an issue, but an announcement and a request for feedback. I do really like the work that you have done over the years to produce astrometry.net. I have been using the package since around 2016 for all of my astrophotography and I have really appreciated how it has improved my sessions. In 2016, I got involved with open source software and started working on the KStars and INDI project and we heavily use Astrometry.net to make it all work. Earlier this year, I started working on trying to improve how Astrometry.net worked with KStars, decided to try using Sextractor to avoid having to have users have python installed properly, found that it worked REALLY well, and started playing around a bit. Then the coronavirus hit and I had some inspirations and slightly expanded the scope of what I was trying to do. I started thinking that it would be nice to have astrometry.net as a library like we have SEP for sextractor and that it would be nice to have it working on windows, and I just kept playing around.

So now, over the last 2 months, I have developed an adaptation of astrometry.net that could be included in an open source QT application along with SEP (sextractor internal library) to plate solve images internally in a program without any need for saving temp files and that will run on Windows and Mac computers as well as on Linux. This development is mainly intended to go into KStars, but could potentially have a broader interest for other open source cross platform astronomy programs running in QT. Please note that this is NOT DONE yet, its still very much a rough draft and needs extensive testing and review. However, right now, it is successfully working on Mac, Windows, and Linux as far as I can tell.

I was wondering if one of you might be willing to take a look at what I have done, point out any issues or mistakes I may have made so that they can be corrected, or make any suggestions you might have for improvement before we incorporate this into the program. Right now, the Libraries are in a tester program that I am using to refine them, optimize settings, and test out plate solving and sextracting images. I am planning to have many people test it out on real images before calling it finished if possible.

Here is my work: https://github.com/rlancaste/sexysolver-tester

Thanks and I really appreciate anything you can do, and ultimately any users that end up using the open source software would too,

Rob Lancaster

[dstndstn]

Hi,

I'm confused about how swapping in source extractor helps with the need for python. Our internal source extraction program is written in C. It's only after source detection that we pass the source lists through "removelines" and "uniformize", which are python programs (that can be turned off with --no-remove-line --uniformize 0).

And it has always worked on Mac -- that's my development platform :) But I hear you that the mix of python.org, homebrew, system, macports, and conda python installations that can be found in the wild is a pain.

If you have changes that would benefit the astrometry.net code, it would be great if you could package them up as a pull request here!

cheers, --dustin

[dstndstn]

ps, the GUI etc looks great and this seems like a really nice packaging effort!

[rlancaste]

Hi Dustin,

Yes, it does work very well on Macs, I have been using it and working with it in KStars since 2016. The issues I have had have to do with distributing KStars to people who don't have homebrew, Macports, or other similar environments on Mac. Or Cygwin/Windows Subsystem for Linux on Windows. A number of end users on Mac or Windows either don't want or don't understand all that complexity and it has been very difficult to help them get set up to do astrophotography. So my goal since then has been to try to make things easier for them to get on board. My ultimate goal is to have the system just work, so that all they have to do is download KStars or a similar program, download the index files that they need inside the program, set the settings inside the program and be good to go.

Sorry if I wasn't totally clear about the purposes of the changes. I am already using the options "--no-remove-line --uniformize 0" settings, and yes they are the thing that helps with getting away from Python. The change that I made toward using sextractor or SEP instead of the internal source extraction helps with getting away from netpbm, which has also been problematic to build with Craft.

We use Craft on Mac to build KStars, INDI, and all required programs. I wrote recipes for astrometry.net and netpbm and other dependencies a couple of years ago and they mostly work, but they are complicated and there have been some problems. Switching to sextractor and using the options "--no-remove-line --uniformize 0" have pretty much eliminated those problems, which I was very happy about. And the side effect that it had of making things much faster, led me to start all this work.

[dstndstn]

Yes, having it work easily out of the box is an admirable goal; nice work!

The internal source detector also only uses FITS, so again, not sure how source extractor helps get away from netpbm :)

I'm surprised that source extractor is any faster -- would be curious what parameters you're using. Are you finding that the source extraction per se is faster, or that the resulting solve is faster? (which would suggest that the quality of the source extraction is better, at least for astrometry.net's purposes)

cheers, --dustin

[rlancaste]

"If you have changes that would benefit the astrometry.net code, it would be great if you could package them up as a pull request here!"

I think there are several categories of the changes I made, some you might want and some you might not.

Some of the changes would probably be good to include for the purposes of improving the code in general, like the elimination of the memory leaks. I think these might be important to include.

Some of them are related to getting it to work on windows natively with both MSVC and MinGW. You might or might not want these changes, I am not sure if you are planning to support Windows natively. In that regard though, If you do want to support this, I think more changes would need to be made to make it fully work because I got away with not making a number of changes because I eliminated code that was not needed since in the internal build, I was not interested in saving lots of temp files, in reading lots of options from the command line, or in a number of features of astrometry.net that we don't need for plate solving within a program like KStars.

Some of the changes you would not want to include at all I think because they have to do with eliminating code to make the internal solver work internally and avoid file system operations. You probably want to keep that code the way it is in the main repo because your target is to produce a command line program that works the way it does now I guess.

Once I get this all finished, I can try to sent some pull requests in the first category, and you can let me know what you think about the others. Would that work?

[rlancaste]

"ps, the GUI etc looks great and this seems like a really nice packaging effort!"

Thanks! Unfortunately, the GUI is just a tester program, the library itself in KStars would not show that GUI! But in doing this I learned some things that I might bring back to the GUI in KStars. the list of star information might be nice for example. Also I am thinking about developing a separate program based on this library that could be useful for doing photometry and light curves automatically. It really is set up well for that purpose.

"Yes, having it work easily out of the box is an admirable goal; nice work!"

Thank you very much. I have been trying to do this for several years now, but building all the netpbm programs, building all the astrometry.net code, and building all the dependencies, and then trying to put them all in the app bundle and package it all up for distribution has been a really challenging thing. So when I started trying to make it internal earlier this year, my goal there was to make things easier for me to get it all packed up for the users. It just turned into something much bigger when coronavirus hit.

[rlancaste]

"I'm surprised that source extractor is any faster -- would be curious what parameters you're using. Are you finding that the source extraction per se is faster, or that the resulting solve is faster? (which would suggest that the quality of the source extraction is better, at least for astrometry.net's purposes)"

I was very surprised by this as well as I did not expect it. I expected that my changes back in January to add an option to use sextractor and the options that avoid python would just simplify my packaging situation and would make it work better for the few people that had trouble getting started. But when the people tested it started telling me that it worked so much faster, it was a pleasant side effect. I don't know why it is faster.

But yes, since then I have spent a bunch of time trying to perfect the settings to make it even faster and it has paid off.

[rlancaste]

"The internal source detector also only uses FITS, so again, not sure how source extractor helps get away from netpbm :)"

To clarify this, yes if the input file is FITS, netpbm is not required, but one feature we have in KStars allows loading a JPEG file or other format file so that the user can plate solve that file and slew to the object to start imaging.

I was led to the solution of using sextractor to avoid netpbm and for using the options to eliminate the python requirements by the "Workarounds" section on your readme: http://astrometry.net/doc/readme.html That was very helpful by the way!

[dstndstn]

Great, would be very happy to have your first category of changes; I'm surprised to hear you found any memory leaks, since I usually test everything with valgrind.

I honestly don't care at all about Windows. :) I guess others do, but, well, I just can't be asked.

It's not like I want to generate a bunch of temp files :) it's just a side effect of gluing together the various pieces of the system to support the various types of inputs and options. In particular, for some time I have wanted to change the way the axy file is generated and written to disk by solve-field and then read in by the exec-d astrometry-engine process; they should share a data structure and solve-field should call a function. But alas, I haven't committed the time to make that happen.

[rlancaste]

"Great, would be very happy to have your first category of changes; I'm surprised to hear you found any memory leaks, since I usually test everything with valgrind."

I found the memory leaks with Xcode's Leak detector. It is possible that the leaks were caused by other changes that I made, so possibly in your code they are not leaks, I don't know. But I think at least a couple of them might be in your code.

"I honestly don't care at all about Windows. :) I guess others do, but, well, I just can't be asked."

Yes I gathered that from your other comments in the forums. I don't personally use windows, but I have many friends who do, and they have really enjoyed using KStars and INDI. They would really appreciate more windows support. Also since KStars is intended for all operating systems, something that I make for Linux or Mac, I should also make for Windows. I did get it working, but perhaps not all of my changes to get it working were the best way to do it. It was just the best I could do locking myself in the basement during the beginning of coronavirus and working through the build and run issues one by one. I don't know since I don't personally use Windows :-)

[rlancaste]

"It's not like I want to generate a bunch of temp files "

Yep, I figured that. But really from the way that astrometry.net is setup to run from the command line and use all of those other programs to do its tasks, you really have to.

I only managed to get away from the temp files because we are loading the image buffer into memory from an image that is either opened by KStars or taken by KStars, then passing a pointer to that buffer to SEP to get the stars, then passing the resulting xylist in memory to my internal build of astrometry.net, and then passing back the resulting SIP object back when the results are found. There are no external programs required. In your setup, you have to deal with a number of programs that need to send their input and output to each other.

The only temp files that I still have to use for the internal build that I made of astrometry.net are the "cancel" file and the "solved" file. In the future I may be able to get rid of those as well, but for now I'm sticking with them. I was even able to eliminate the astrometry.net config file.

[dstndstn]

Yeah, that's all similar to the "control-program.c" example situation.

[dstndstn]

(and it would all be structured very differently if I started writing it today! But in the direction of way MORE python centric.)

[rlancaste]

"MORE python centric".

Oh no, not more python! It would be fine if OS X had a standard install of python 3 that worked for everybody, but as it stands now, so many people have so many different setups that its crazy

[rlancaste]

Ah, control-program.c does have some similarities to what I did inside two methods of my internal solver in the library. I don't think I had noticed that one. I was basing what I did upon augment-xylist-main and engine-main instead of on control-program. That is very helpful. I will take a look at that and see if there is anything that I missed.

[dstndstn]

the docs need a serious updating & cleanup!

[dstndstn]

(as far as I can tell, this is basically always true for all projects)

[rlancaste]

oh yes I understand. My documentation always leaves a lot to be desired. That is ok. In creating my own solution from augment-xylist-main and engine-main, I think I learned a lot more than if I had based it on control-program :-) .

And my library is much more than just the part that resembles control-program. It is meant to be a general interface to the internal SEP and internal solver, to any external sextractor or astrometry.net, and the online versions on nova.astrometry.net. Plus, I got the internal build working on windows too.

I will review that file and hopefully make this even better :-). thanks for pointing it out.

[rlancaste]

Oh, I just remembered another python issue. Yes it would be nice if OS X had a standard Python 3 installation, but if it is anything like the current python 2 installation, there would still be a number of issues. Several years ago, I had tried to have users use the default python 2 installation, but then either they or the software had to install numpy, and astropy or pyfits, but that caused all kinds of problems since installing packages in the default python installation requires admin access. And sometimes when they did this and they had another python installed in a different location, there were issues where the packages were installed in one and not the other.

So even if there was a standard Python3, that would not be the end of my Python headaches.

[dstndstn]

Yes, this is a large issue. One is tempted to turn to Docker :) but of course that doesn't help for graphical apps

[wtgee]

Sorry, I haven't followed the entire thread, but @rlancaste the standard way to do all these things these days would be with docker and/or anaconda (or something similar) rather than trying to alter all your dependencies.

It's fairly trivial to install astromtery.net in a docker image and have it work cross-platform.

People have recently been reporting success on the anaconda version (#138)

Edit: oops, I see @dstndstn just mentioned the docker route...

[rlancaste]

Yes, I understand that docker is quite useful on Linux. I think that Jasem is planning to do some things with docker in order to make INDI easier to install and use. I also understand that it can be nice on Mac OS and windows for command line programs. I don't have much experience with docker, but I have heard that it helps in these areas.

But that is really not my end goal. I have mainly been involved with KStars and INDI. I ported KStars to Macs to get it to work as an App that can be installed by simply dragging and dropping it to the Applications folder, and have tried to make it as easy to use as possible for users that don't know the command line. KStars on all platforms, windows, Mac, and Linux depends upon astrometry.net and other programs in order to perform a number of functions. Installing these programs and configuring them can be complex, plus the fact that they are all really command line programs.

Docker can make it easier, but I am interested in having the user be able to just drag and drop the KStars app bundle to install it and have everything work right away without having to do lots of extra work. I don't believe Docker can do that. We already have homebrew on Mac and Cygwin/ANSVR on Linux and they both install astrometry.net very well and we have used them for years to make KStars work on different platforms. I think docker would just replace that method of installing the command line version of astrometry.net, correct? Unless you can bundle the docker programs inside the app bundle, and have them just work with python and all required dependencies also bundled inside (ie no command line work from the user), that wouldn't solve my problems.

The other point is that a significant part of my goal is to try to get away from all the saving and opening of files, the temporary files, the configuration files, all the dependencies, etc. I would rather keep the image buffer in memory and just plate solve it and get the result back inside the program without writing anything to disk at all.

[rlancaste]

Part of my library that I have produced is the internal solver, but part of it is to support the use of an external solver a user has installed on their computer like an external ANSVR or astrometry.net. Docker might be able to help in this area perhaps.

[rlancaste]

It's funny that you mentioned Documentation Dustin,

When I wrote the documentation on this site: https://github.com/rlancaste/sexysolver-tester. which actually wasn't that long ago, the jpeg image at the top of the page took a little less than 30 seconds to blind solve. Today, if I blind solve that same image with my library, it takes 0.87 seconds to blind solve. Note that that time includes the time it takes to extract the sources and to solve the image.

Also since that documentation was written, I made some major changes to the interface and the class organization. Plus now it can do a sextraction based on a prior solve since the WCS information was retained, and thus extracted stars can have RA and DEC as well. So yes, I need to update the documentation too. So I don't blame you for out of date documentation.