astrometry.net address hardcoded, impossible to use a local running nova server

[papaf76]

Hi, looking here: https://github.com/rlancaste/stellarsolver/blob/master/stellarsolver/onlinesolver.cpp#L325 It seems the software is using a hardcoded value of nova.astrometry.net in place of the specified server (eg. from ekos config). This makes it impossible to use a local copy of nova.

[rlancaste]

One of the main reasons for starting StellarSolver was originally to get rid of python. In fact those exact three parameters are passed to local astrometry and Cygwin and ANSVR most of the time. It is only when the user wants to do the traditional “built in” solver for astrometry.net that astrometry is allowed to solve it how it wants.

[laheller]

@rlancaste

According to astrometry.net documentation python is used only in two cases:

1. When input image is NOT the FITS format => converts it using python scripts to FITS
2. When user wants to plot annotations on the solved input image (other than FITS) => annotations are done by a python script

In our astrophotography scenarios none of the above is the case. We simply use FITS format and do NOT plot. Next to just start another new python3 process by solve-field, without using its result is simply wasting of resources and it just slows down the overall plate solving.

So as you wrote:

One of the main reasons for starting StellarSolver was originally to get rid of python.

If that's the main reason, then why in some cases python is allowed when it's anyway not needed? :-)

BR,

Ladislav

[rlancaste]

To be honest, the main reason I left astrometry.net to its own devices in that case was so that we could have a basis for comparison. The main goal was to use Sextractor or SEP combined with astrometry.net to avoid needing any python or netpbm at all. And when I did my first experiments, that method was giving far superior results to the astrometry internal star extraction and solving. I definitely see it as the preferred method in almost all cases, including using online astrometry.net or remote ANSVR, because then you are uploading a text file instead of an image file, its WAY smaller. The only reasons I left in the ability to solve using an image and astrometry's built in methods was so that we could compare the performance to what astrometry naturally wants to do and also so that people would still be able to use astrometry.net if for some reason SEP or Sextractor didn't work for them and astrometry.net's method worked better. But so far, SEP and sextractor still work much better almost universally.

Actually one of the best abilities of StellarSolver is its ability to plate solve jpeg images very very fast. To compare its performance to using astrometry.net natively, we would need to allow it to plate solve using python to do the conversion which is what it naturally does. If we are converting it to FITS first, that isn't normal astrometry behavior, so it isn't really a good comparison. Now I could see adding an option that allows the user to force all images to be converted to FITS before being sent to astrometry.net, but that is adding complexity.

It is true that many times we are working with FITS images in KStars, but the "load and slew" feature often works with Jpeg images. Those Jpegs would need to be converted before being sent.

[rlancaste]

Let me think about this, because we could either convert all images to FITS first before sending them to StellarSolver, or I could add another option to StellarSolver for always converting to FITS first. We could hard code that as turned on in KStars and then we could remove all references to Python from the Align options. That also would allow me to delete the crazy recipe's in the KStars Mac craft build that require the building of netpbm and the inclusion of python. That might be a nice idea. KStars does not need to be sending JPEGS to astrometry if we an avoid it. Good idea!

[rlancaste]

Ok I made the python avoidance commands universal and I made an option that would only sent FITS files to astrometry.

The next step will be to add the usage of this feature to Ekos. Then hopefully we can get rid of some unnecessary recipes in craft and remove clutter from KStars. Maybe this will end our astrometry python issues for good. Actually I thought I had done that, but they came back in this thread. Hopefully now. . .

There will be a few steps before it will get into the windows and Mac releases.

[laheller]

@rlancaste

Just made a short test, how python affects the overall plate solving performance. I used the free tool called ProcMon, which is able to monitor running processes on Windows.

Configured its filter to allow monitor only solve-field, shell processes and ProcessCreate operation:

First I executed a plate-solving (in the Cygwin terminal) of a PNG image, where also python is involved at least by converting PNG to FITS via netpbm. I also let solve-field to plot annotations. Command line in Cygwin was:

solve-field -O -l 30 -D ./Solve -L 40 -H 90 -u dw -z 3 ./M31.png

In ProcMon we can see, that solve-field creates a lot of subprocesses including among others python:

As we can see in the monitor, python was 3 times called.

In the second case I used again a PNG image, but this time with following command line:

solve-field -O -l 30 -D ./Solve -L 40 -H 90 -u dw -z 3 -9 --uniformize 0 -p ./M31.png

From the output we can clearly see that python was called only once to do PNG->FITS conversion:

Finally in the third case I used FITS input image with additional command line parameters to completely skip python:

solve-field -O -l 30 -D ./Solve -L 40 -H 90 -u dw -z 3 -9 --uniformize 0 -p -y --fits-image ./M31.fits

Please note that the --fits-image parameter is also required to reach our goal! When this parameter is not specified for a FITS input file, python is still called! Just checked your already pushed commit and saw that this parameter is mising ;-)

As we expected, the python was not called by solver and the overall plate-solving was much faster:

Performance and the fact that we don't (really) need python are the only reason why I suggest to improve StellarSolve by addig the above command line parameters!

BR,

Ladislav

[laheller]

@rlancaste

In the meantime I tried again with my local custom built Cygwin/astrometry.net using this combination:

Means: Solving method = Local Astrometry Source Extraction Method = Internal SEP

Using the above combination plate-solving works well, here is the output:

2021-08-24T21:06:14 Solver RA (296.14873) DEC (9.83861) Orientation (-178.26570) Pixel Scale (124.45108) Parity (neg)
2021-08-24T21:06:14 Solver completed after 1.43 seconds.
2021-08-24T21:06:14 Shutting down other child solvers
2021-08-24T21:06:14 Successfully solved with child solver: 3
2021-08-24T21:06:14 Finished Loading WCS...
2021-08-24T21:06:14 Loading WCS from file...
2021-08-24T21:06:14 Field parity: neg

2021-08-24T21:06:14 Field rotation angle: up is -178.266 degrees E of N
2021-08-24T21:06:14 Pixel Scale: 124.451"
2021-08-24T21:06:14 Field size: 63.31 x 44.84 arcminutes
2021-08-24T21:06:14 Field center: (RA H:M:S, Dec D:M:S) = (19:44:35.694, +09:50:18.994).
2021-08-24T21:06:14 Field center: (RA,Dec) = (296.149, 9.83861) deg.
2021-08-24T21:06:14 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
2021-08-24T21:06:13 Command: C:/astrometry.net/bin/solve-field.exe -O --no-plots --no-verify --crpix-center --match none --corr none --new-fits none --rdls none --resort --objs 50 --odds-to-solve 1e+09 --odds-to-tune-up 1e+06 -L 20.625 -H 52.5 -u degwidth --width 2048 --height 1364 --x-column X_IMAGE --y-column Y_IMAGE --sort-column MAG_AUTO --sort-ascending --no-remove-lines --uniformize 0 --backend-config C:/Users/Ladislav Heller/AppData/Local/Temp/externalSextractorSolver_1_1.cfg --cancel C:/Users/Ladislav Heller/AppData/Local/Temp/externalSextractorSolver_1_1.cancel -W C:/Users/Ladislav Heller/AppData/Local/Temp/externalSextractorSolver_1_1.wcs C:/Users/Ladislav Heller/AppData/Local/Temp/externalSextractorSolver_1_1.xyls
2021-08-24T21:06:13 Starting external Astrometry.net solver with the 3-LargeScaleSolving profile...
2021-08-24T21:06:13 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
2021-08-24T21:06:13 Command: C:/astrometry.net/bin/solve-field.exe -O --no-plots --no-verify --crpix-center --match none --corr none --new-fits none --rdls none --resort --objs 50 --odds-to-solve 1e+09 --odds-to-tune-up 1e+06 -L 105.625 -H 180 -u degwidth --width 2048 --height 1364 --x-column X_IMAGE --y-column Y_IMAGE --sort-column MAG_AUTO --sort-ascending --no-remove-lines --uniformize 0 --backend-config C:/Users/Ladislav Heller/AppData/Local/Temp/externalSextractorSolver_1_3.cfg --cancel C:/Users/Ladislav Heller/AppData/Local/Temp/externalSextractorSolver_1_3.cancel -W C:/Users/Ladislav Heller/AppData/Local/Temp/externalSextractorSolver_1_3.wcs C:/Users/Ladislav Heller/AppData/Local/Temp/externalSextractorSolver_1_3.xyls
2021-08-24T21:06:13 Starting external Astrometry.net solver with the 3-LargeScaleSolving profile...
2021-08-24T21:06:13 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
2021-08-24T21:06:13 Command: C:/astrometry.net/bin/solve-field.exe -O --no-plots --no-verify --crpix-center --match none --corr none --new-fits none --rdls none --resort --objs 50 --odds-to-solve 1e+09 --odds-to-tune-up 1e+06 -L 52.5 -H 105.625 -u degwidth --width 2048 --height 1364 --x-column X_IMAGE --y-column Y_IMAGE --sort-column MAG_AUTO --sort-ascending --no-remove-lines --uniformize 0 --backend-config C:/Users/Ladislav Heller/AppData/Local/Temp/externalSextractorSolver_1_2.cfg --cancel C:/Users/Ladislav Heller/AppData/Local/Temp/externalSextractorSolver_1_2.cancel -W C:/Users/Ladislav Heller/AppData/Local/Temp/externalSextractorSolver_1_2.wcs C:/Users/Ladislav Heller/AppData/Local/Temp/externalSextractorSolver_1_2.xyls
2021-08-24T21:06:13 Command: C:/astrometry.net/bin/solve-field.exe -O --no-plots --no-verify --crpix-center --match none --corr none --new-fits none --rdls none --resort --objs 50 --odds-to-solve 1e+09 --odds-to-tune-up 1e+06 -L 10 -H 20.625 -u degwidth --width 2048 --height 1364 --x-column X_IMAGE --y-column Y_IMAGE --sort-column MAG_AUTO --sort-ascending --no-remove-lines --uniformize 0 --backend-config C:/Users/Ladislav Heller/AppData/Local/Temp/externalSextractorSolver_1_0.cfg --cancel C:/Users/Ladislav Heller/AppData/Local/Temp/externalSextractorSolver_1_0.cancel -W C:/Users/Ladislav Heller/AppData/Local/Temp/externalSextractorSolver_1_0.wcs C:/Users/Ladislav Heller/AppData/Local/Temp/externalSextractorSolver_1_0.xyls
2021-08-24T21:06:13 Starting external Astrometry.net solver with the 3-LargeScaleSolving profile...
2021-08-24T21:06:13 Starting external Astrometry.net solver with the 3-LargeScaleSolving profile...
2021-08-24T21:06:13 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
2021-08-24T21:06:13 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
2021-08-24T21:06:13 Configuring external Astrometry.net solver using an xylist input
2021-08-24T21:06:13 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
2021-08-24T21:06:13 Configuring external Astrometry.net solver using an xylist input
2021-08-24T21:06:13 The external ANSVR solver on windows does not handle the inparallel option well, disabling it for this run.
2021-08-24T21:06:13 Configuring external Astrometry.net solver using an xylist input
2021-08-24T21:06:13 Configuring external Astrometry.net solver using an xylist input
2021-08-24T21:06:13 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
2021-08-24T21:06:13 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
2021-08-24T21:06:13 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
2021-08-24T21:06:13 The external ANSVR solver on windows does not handle the inparallel option well, disabling it for this run.
2021-08-24T21:06:13 The external ANSVR solver on windows does not handle the inparallel option well, disabling it for this run.
2021-08-24T21:06:13 The external ANSVR solver on windows does not handle the inparallel option well, disabling it for this run.
2021-08-24T21:06:12 Solver # 4, Low 105.625, High 180 arcminwidth
2021-08-24T21:06:12 Solver # 3, Low 52.5, High 105.625 arcminwidth
2021-08-24T21:06:12 Solver # 2, Low 20.625, High 52.5 arcminwidth
2021-08-24T21:06:12 Solver # 1, Low 10, High 20.625 arcminwidth
2021-08-24T21:06:12 Starting 4 threads to solve on multiple scales
2021-08-24T21:06:12 Stars Found after Filtering: 50
2021-08-24T21:06:12 Keeping just the 50 brightest stars
2021-08-24T21:06:12 Removing the stars with a/b ratios greater than 1.5
2021-08-24T21:06:12 Stars Found before Filtering: 497
2021-08-24T21:06:12 Starting Internal StellarSolver Sextractor with the 3-LargeScaleSolving profile . . .
2021-08-24T21:06:12 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
2021-08-24T21:06:12 There should be enough RAM to load the indexes in parallel.
2021-08-24T21:06:12 Evaluating Installed RAM for inParallel Option.  Total Size of Index files: 0.634597 GB, Installed RAM: 15.9458 GB, Free RAM: 10.352 GB

.

Tried again also the combination: Solving method = Local Astrometry Source Extraction Method = BuiltIn method for solver It again failed like many times before, but this time I took the command line from log, as you did and put it to the Cygwin terminal and started and guess what happend: It works from Cygwin! Can't believe, but the same command line works from Cygwin terminal!!!

I will do further investingating, but now it is clear that my local custom Cywin/astrometry.net is fully working, but somehow when started from StellarSolver, it fails for the above combination.

BR,

Ladislav

[rlancaste]

if you think the --fits-image parameter is really needed when a user wants to use the "built in" star extraction in order to avoid python. I am not sure why it would need that when its being given a FITS file already? maybe it isn't sure it is a FITS file till it tries?

[rlancaste]

For your experiments, I don't think you want the "large-scale solving" profile, which I see selected in your screenshot. I designed that one to work best for DSLR lens scale images. You probably want "Parallel small scale"

[rlancaste]

When I checked your cygwin build it had a broken link as I said, when I fixed that it worked fine.

[laheller]

if you think the --fits-image parameter is really needed when a user wants to use the "built in" star extraction in order to avoid python. I am not sure why it would need that when its being given a FITS file already? maybe it isn't sure it is a FITS file till it tries?

@rlancaste Just try it with and without this parameter, while watching subprocesses in ProcMon tool, if you don't trust me :) In my environment python was only skipped when input file was a FITS and following parameters were passed to solve-field:

• -p (--no-plots)
• -9 ( --no-remove-lines)
• --uniformize 0
• --fits-image

Any other combination of input format and command line parameters involved python.

BR,

Ladislav

[laheller]

When I checked your cygwin build it had a broken link as I said, when I fixed that it worked fine.

@rlancaste Yes I know, but in my cygwin build that symbolic link is correct! Anyway, plate-solve failed.

BR,

Ladislav

[laheller]

For your experiments, I don't think you want the "large-scale solving" profile, which I see selected in your screenshot. I designed that one to work best for DSLR lens scale images. You probably want "Parallel small scale"

@rlancaste Tried all 4 default profiles and the builtin extractor but always the same error:

2021-08-24T22:57:33 Solver Failed.
2021-08-24T22:57:33 Child solver: 2 did not solve or was aborted
2021-08-24T22:57:33 Child solver: 4 did not solve or was aborted
2021-08-24T22:57:33 Child solver: 1 did not solve or was aborted
2021-08-24T22:57:33 Child solver: 3 did not solve or was aborted
2021-08-24T22:57:32 Command: C:/astrometry.net/bin/solve-field.exe -O --no-plots --no-verify --crpix-center --match none --corr none --new-fits none --rdls none --resort --objs 50 --downsample 2 --odds-to-solve 1e+09 --odds-to-tune-up 1e+06 -L 0.71875 -H 2.575 -u degwidth --backend-config C:/Users/Ladislav Heller/AppData/Local/Temp/externalSextractorSolver_1_1.cfg --cancel C:/Users/Ladislav Heller/AppData/Local/Temp/externalSextractorSolver_1_1.cancel -W C:/Users/Ladislav Heller/AppData/Local/Temp/externalSextractorSolver_1_1.wcs --keep-xylist C:/Users/Ladislav Heller/AppData/Local/Temp/externalSextractorSolver_1_1.xyls C:/Users/Ladislav Heller/AppData/Local/Temp/externalSextractorSolver_1_1.fits
2021-08-24T22:57:32 Starting external Astrometry.net solver with the 4-SmallScaleSolving profile...
2021-08-24T22:57:32 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
2021-08-24T22:57:32 Command: C:/astrometry.net/bin/solve-field.exe -O --no-plots --no-verify --crpix-center --match none --corr none --new-fits none --rdls none --resort --objs 50 --downsample 2 --odds-to-solve 1e+09 --odds-to-tune-up 1e+06 -L 2.575 -H 5.66875 -u degwidth --backend-config C:/Users/Ladislav Heller/AppData/Local/Temp/externalSextractorSolver_1_2.cfg --cancel C:/Users/Ladislav Heller/AppData/Local/Temp/externalSextractorSolver_1_2.cancel -W C:/Users/Ladislav Heller/AppData/Local/Temp/externalSextractorSolver_1_2.wcs --keep-xylist C:/Users/Ladislav Heller/AppData/Local/Temp/externalSextractorSolver_1_2.xyls C:/Users/Ladislav Heller/AppData/Local/Temp/externalSextractorSolver_1_2.fits
2021-08-24T22:57:32 Starting external Astrometry.net solver with the 4-SmallScaleSolving profile...
2021-08-24T22:57:32 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
2021-08-24T22:57:32 Command: C:/astrometry.net/bin/solve-field.exe -O --no-plots --no-verify --crpix-center --match none --corr none --new-fits none --rdls none --resort --objs 50 --downsample 2 --odds-to-solve 1e+09 --odds-to-tune-up 1e+06 -L 0.1 -H 0.71875 -u degwidth --backend-config C:/Users/Ladislav Heller/AppData/Local/Temp/externalSextractorSolver_1_0.cfg --cancel C:/Users/Ladislav Heller/AppData/Local/Temp/externalSextractorSolver_1_0.cancel -W C:/Users/Ladislav Heller/AppData/Local/Temp/externalSextractorSolver_1_0.wcs --keep-xylist C:/Users/Ladislav Heller/AppData/Local/Temp/externalSextractorSolver_1_0.xyls C:/Users/Ladislav Heller/AppData/Local/Temp/externalSextractorSolver_1_0.fits
2021-08-24T22:57:32 Command: C:/astrometry.net/bin/solve-field.exe -O --no-plots --no-verify --crpix-center --match none --corr none --new-fits none --rdls none --resort --objs 50 --downsample 2 --odds-to-solve 1e+09 --odds-to-tune-up 1e+06 -L 5.66875 -H 10 -u degwidth --backend-config C:/Users/Ladislav Heller/AppData/Local/Temp/externalSextractorSolver_1_3.cfg --cancel C:/Users/Ladislav Heller/AppData/Local/Temp/externalSextractorSolver_1_3.cancel -W C:/Users/Ladislav Heller/AppData/Local/Temp/externalSextractorSolver_1_3.wcs --keep-xylist C:/Users/Ladislav Heller/AppData/Local/Temp/externalSextractorSolver_1_3.xyls C:/Users/Ladislav Heller/AppData/Local/Temp/externalSextractorSolver_1_3.fits
2021-08-24T22:57:32 Starting external Astrometry.net solver with the 4-SmallScaleSolving profile...
2021-08-24T22:57:32 Starting external Astrometry.net solver with the 4-SmallScaleSolving profile...
2021-08-24T22:57:32 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
2021-08-24T22:57:32 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
2021-08-24T22:57:32 Configuring external Astrometry.net solver classically: using python and without Sextractor first
2021-08-24T22:57:32 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
2021-08-24T22:57:32 The external ANSVR solver on windows does not handle the inparallel option well, disabling it for this run.
2021-08-24T22:57:32 Configuring external Astrometry.net solver classically: using python and without Sextractor first
2021-08-24T22:57:32 Configuring external Astrometry.net solver classically: using python and without Sextractor first
2021-08-24T22:57:32 Configuring external Astrometry.net solver classically: using python and without Sextractor first
2021-08-24T22:57:32 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
2021-08-24T22:57:32 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
2021-08-24T22:57:32 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
2021-08-24T22:57:32 The external ANSVR solver on windows does not handle the inparallel option well, disabling it for this run.
2021-08-24T22:57:32 The external ANSVR solver on windows does not handle the inparallel option well, disabling it for this run.
2021-08-24T22:57:32 The external ANSVR solver on windows does not handle the inparallel option well, disabling it for this run.
2021-08-24T22:57:32 Solver # 4, Low 5.66875, High 10 arcminwidth
2021-08-24T22:57:32 Solver # 3, Low 2.575, High 5.66875 arcminwidth
2021-08-24T22:57:32 Solver # 2, Low 0.71875, High 2.575 arcminwidth
2021-08-24T22:57:32 Solver # 1, Low 0.1, High 0.71875 arcminwidth
2021-08-24T22:57:32 Starting 4 threads to solve on multiple scales
2021-08-24T22:57:32 There should be enough RAM to load the indexes in parallel.
2021-08-24T22:57:32 Evaluating Installed RAM for inParallel Option.  Total Size of Index files: 0.634597 GB, Installed RAM: 15.9458 GB, Free RAM: 9.86173 GB
2021-08-24T22:57:32 Automatically downsampling the image by 2

BR,

Ladislav

[rlancaste]

Ok I added the --fits-image parameter

[rlancaste]

Ok I think I should close this issue here now since we pretty much resolved both issues in the thread.

[rlancaste]

The next release of KStars should get these changes in it.

[laheller]

Ok I think I should close this issue here now since we pretty much resolved both issues in the thread.

OK for me.

The next release of KStars should get these changes in it.

Great!