astrometry.net vs. Android

[laheller]

Hi @dstndstn

I have a very specific issue. I just build astrometry.net (including all its dependencies, except python) for the Android Aarch64 platform using the latest Android NDK. Yes, it is possible to build it for Android, however, it's a bit tricky and takes a while! Long story short, I copied all the built stuff and some index files to my Samsung Galaxy S20+ rooted device to test it in the Terminal application. Tried plate-solving on one test fits image but unfortunately, it failed at the astrometry-engine tool. Only the augmented XY file was created. Then I just tried only the astrometry-engine tool with that AXY file and only one index file like this:

astrometry-engine -v -c none -i ./astrometry.net/data/index-4118.fits ./Solve/Sas.axy

The result is always the same, no matter, how the astrometry-engine was called:

Index name "./astrometry.net/data/index-4118.fits" is readable, using as index filename
Index name "./astrometry.net/data/index-4118.fits" is readable, using as index filename
Loading metadata for ./astrometry.net/data/index-4118.fits...
Index name "./astrometry.net/data/index-4118.fits" is readable, using as index filename
Index scale: [1000, 1400] arcmin, [60000, 84000] arcsec
Index has 3072 quads and 1920 stars
Reading file "./Solve/Sas.axy"...
Set odds ratio to solve to 1e+09 (log = 20.7233)
Couldn't allocate memory for a bl node!
Segmentation fault

On my device, I have 8 GB of RAM and at the moment I tried the above, I had more than 4 GB free. So my question is, why the astrometry-engine can't allocate memory for its whatever stuff? Is there any way, trick, command-line switch, or whatever to workaround the above?

When needed, I can upload somewhere the complete build of astrometry.net and dependencies, means all the binaries, if you want to try it.

BR,

Ladislav

[dstndstn]

Hi,

Is there any way you can run it in a debugger? Or, ideally, valgrind. It is very difficult to say where that error is coming from. ("bl" is the "block-list" array data structure that we use throughout the code for growable lists of things.) The astrometry.net code is CI tested on ARM Linux, for whatever that is worth. It works on Linux, MacOS and Solaris, and clang and gcc and icc compilers. So my guess is that your compiler / linker / toolchain is doing something weird. I have no experience with Android so wouldn't know where to start in tracking this down.

[laheller]

@dstndstn

For now the only thing I figured out that this error message comes from: https://github.com/dstndstn/astrometry.net/blob/4b227b6b70befa6fa4fa7ee154d719c60236d65b/util/bl.c#L396

The toolchain in Android NDK latest version is LLVM and has clang compiler. By debug you mean compile everything using -gXXX and use lldb?

[dstndstn]

Sure, yes, try compiling with debug symbols and run astrometry-engine inside the debugger to try to figure out what call stack that error is coming from. The Astrometry.net Makefile should pass debugging flags to the compiler if you do make OPTIMIZE=no

[laheller]

@dstndstn

Very interesting, I just rebuilt the whole astrometry.net using the below variables:

CFLAGS="... -ggdb3" OPTIMIZE=no ARCH_FLAGS="-march=armv8.3a -mtune=generic"

Not sure, which one of the above helped, but the error is now GONE and astrometry-engine does its job without problems!!! Now we can plate-solve on Android devices! It's time to build a nice GUI app around the binaries :)

BTW, I had to hack a little bit util/Makefile because a rule defined there prevents to include netpbm for Android into astrometry.net even when netpbm CFLAGS/LIBS are correctly defined. Without modifying that file the test netpbm binary however is built and the build process tries to start it which obviously fails due to invalid exec format.

Anyway, following tools were tested from the suite on a Samsung Galaxy S20+ Android device:

• solve-field
• astrometry-engine
• wcsinfo
• plot-constellations

All of them works perfectly!

For reference here is the full script I am using to build astrometry.net:

#!/bin/sh

export ANDROID_NDK=${HOME}/android-ndk-r23b
export TOOLCHAIN=llvm
export TARGET=aarch64-linux-android
export API=29
export ANDROID_PREFIX=${ANDROID_NDK}/toolchains/${TOOLCHAIN}/prebuilt/linux-x86_64
export SYSROOT=${ANDROID_PREFIX}/sysroot

export BIN_PATH=${ANDROID_PREFIX}/bin
export CROSS_PATH=${ANDROID_PREFIX}/bin/${TARGET}

export AR=${BIN_PATH}/${TOOLCHAIN}-ar
export AS=${CROSS_PATH}-as
export CC=${CROSS_PATH}${API}-clang
export CXX=${CROSS_PATH}${API}-clang++
export LD=${BIN_PATH}/ld
export NM=${BIN_PATH}/${TOOLCHAIN}-nm
export OBJDUMP=${BIN_PATH}/${TOOLCHAIN}-objdump
export RANLIB=${BIN_PATH}/${TOOLCHAIN}-ranlib
export STRIP=${BIN_PATH}/${TOOLCHAIN}-strip

export CFLAGS="${CFLAGS} --sysroot=${SYSROOT} -I${SYSROOT}/usr/include -I${HOME}/repos/bzip2/_deploy_aarch64/include -I${HOME}/repos/netpbm/_deploy_aarch64/include -I${HOME}/repos/netpbm/_deploy_aarch64/include/netpbm -fPIE -fPIC -std=c17 -ggdb3"
export CPPFLAGS="${CFLAGS}"
export LDFLAGS="${LDFLAGS} --sysroot=${SYSROOT} -L${SYSROOT}/usr/lib -lm -L${HOME}/repos/bzip2/_deploy_aarch64/lib -lbz2 -L${HOME}/repos/netpbm/_deploy_aarch64/sharedlink -lnetpbm -pie"

export OPTIMIZE=no
export ARCH_FLAGS="-march=armv8.3a -mtune=generic"

export CAIRO_INC="-I${HOME}/repos/cairo/_deploy_aarch64/include/cairo"
export CAIRO_LIB="-L${HOME}/repos/cairo/_deploy_aarch64/lib -lcairo"

export CFITS_INC="-I${HOME}/repos/cfitsio-4.0.0/_deploy_aarch64/include"
export CFITS_LIB="-L${HOME}/repos/cfitsio-4.0.0/_deploy_aarch64/lib -lcfitsio"

export SYSTEM_GSL=yes
export GSL_INC="-I${HOME}/repos/gsl/_deploy_aarch64/include"
export GSL_LIB="-L${HOME}/repos/gsl/_deploy_aarch64/lib -lgsl -lgslcblas"

export JPEG_INC="-I${HOME}/repos/libjpeg-turbo/_deploy_aarch64/include"
export JPEG_LIB="-L${HOME}/repos/libjpeg-turbo/_deploy_aarch64/lib -ljpeg"

export PNG_INC="-I${HOME}/repos/libpng/_deploy_aarch64/include/libpng16"
export PNG_LIB="-L${HOME}/repos/libpng/_deploy_aarch64/lib -lpng16"

export WCSLIB_INC="-I${HOME}/repos/wcslib-7.7/_deploy_aarch64/include/wcslib"
export WCSLIB_LIB="-L${HOME}/repos/wcslib-7.7/_deploy_aarch64/lib -lwcs"

export INSTALL_DIR=$(pwd)/_install

make
make extra
make install

BR,

Ladislav

[dstndstn]

Great!

Would you mind giving details about the util/Makefile change, or submit a PR? May as well fix it if it's a generic bug.

On Wed, Dec 8, 2021 at 3:12 PM Ladislav Heller @.***> wrote:

@dstndstn https://github.com/dstndstn

Very interesting, I just rebuilt the whole astrometry.net using the below variables:

CFLAGS="... -ggdb3" OPTIMIZE=no ARCH_FLAGS="-march=armv8.3a -mtune=generic"

Not sure, which one of the above helped, but the error is now GONE and astrometry-engine does its job without problems!!! Now we can plate-solve on Android devices! It's time to build a nice GUI app around the binaries :)

BTW, I had to hack a little bit util/Makefile because a rule defined there prevents to include netpbm for Android into astrometry.net even when netpbm CFLAGS/LIBS are correctly defined. Without modifying that file the test netpbm binary however is built, but the build process tries to start it which obviously fails due to invalid exec format.

Anyway, following tools were tested from the suite on Android:

• solve-field
• astrometry-engine
• wcsinfo
• plot-constellations

All of them works perfectly!

For reference here is the full script I am using to build astrometry.net http://astrometry.net:

!/bin/sh

export ANDROID_NDK=${HOME}/android-ndk-r23b export TOOLCHAIN=llvm export TARGET=aarch64-linux-android export API=29 export ANDROID_PREFIX=${ANDROID_NDK}/toolchains/${TOOLCHAIN}/prebuilt/linux-x86_64 export SYSROOT=${ANDROID_PREFIX}/sysroot

export BIN_PATH=${ANDROID_PREFIX}/bin export CROSS_PATH=${ANDROID_PREFIX}/bin/${TARGET}

export AR=${BIN_PATH}/${TOOLCHAIN}-ar export AS=${CROSS_PATH}-as export CC=${CROSS_PATH}${API}-clang export CXX=${CROSS_PATH}${API}-clang++ export LD=${BIN_PATH}/ld export NM=${BIN_PATH}/${TOOLCHAIN}-nm export OBJDUMP=${BIN_PATH}/${TOOLCHAIN}-objdump export RANLIB=${BIN_PATH}/${TOOLCHAIN}-ranlib export STRIP=${BIN_PATH}/${TOOLCHAIN}-strip

export CFLAGS="${CFLAGS} --sysroot=${SYSROOT} -I${SYSROOT}/usr/include -I${HOME}/repos/bzip2/_deploy_aarch64/include -I${HOME}/repos/netpbm/_deploy_aarch64/include -I${HOME}/repos/netpbm/_deploy_aarch64/include/netpbm -fPIE -fPIC -std=c17 -ggdb3" export CPPFLAGS="${CFLAGS}" export LDFLAGS="${LDFLAGS} --sysroot=${SYSROOT} -L${SYSROOT}/usr/lib -lm -L${HOME}/repos/bzip2/_deploy_aarch64/lib -lbz2 -L${HOME}/repos/netpbm/_deploy_aarch64/sharedlink -lnetpbm -pie"

export OPTIMIZE=no export ARCH_FLAGS="-march=armv8.3a -mtune=generic"

export CAIRO_INC="-I${HOME}/repos/cairo/_deploy_aarch64/include/cairo" export CAIRO_LIB="-L${HOME}/repos/cairo/_deploy_aarch64/lib -lcairo"

export CFITS_INC="-I${HOME}/repos/cfitsio-4.0.0/_deploy_aarch64/include" export CFITS_LIB="-L${HOME}/repos/cfitsio-4.0.0/_deploy_aarch64/lib -lcfitsio"

export SYSTEM_GSL=yes export GSL_INC="-I${HOME}/repos/gsl/_deploy_aarch64/include" export GSL_LIB="-L${HOME}/repos/gsl/_deploy_aarch64/lib -lgsl -lgslcblas"

export JPEG_INC="-I${HOME}/repos/libjpeg-turbo/_deploy_aarch64/include" export JPEG_LIB="-L${HOME}/repos/libjpeg-turbo/_deploy_aarch64/lib -ljpeg"

export PNG_INC="-I${HOME}/repos/libpng/_deploy_aarch64/include/libpng16" export PNG_LIB="-L${HOME}/repos/libpng/_deploy_aarch64/lib -lpng16"

export WCSLIB_INC="-I${HOME}/repos/wcslib-7.7/_deploy_aarch64/include/wcslib" export WCSLIB_LIB="-L${HOME}/repos/wcslib-7.7/_deploy_aarch64/lib -lwcs"

export INSTALL_DIR=$(pwd)/_install

make make extra make install

BR,

Ladislav

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/dstndstn/astrometry.net/issues/246#issuecomment-989159765, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAIEH7MJMI6G6TLXUXDV6ITUP63ZFANCNFSM5JFVM4TQ . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

[laheller]

@dstndstn

It's a dirty hack simply to get the things working on my system, but shorty I replaced the call of the binary ./os-features-test-netpbm (and redirection) here: https://github.com/dstndstn/astrometry.net/blob/4b227b6b70befa6fa4fa7ee154d719c60236d65b/util/Makefile#L272

and here: https://github.com/dstndstn/astrometry.net/blob/4b227b6b70befa6fa4fa7ee154d719c60236d65b/util/Makefile#L299

with echo ""

With the above hack the test was evaluated to True and _HAVENETPBM was set to Yes.

A good solution to the above case would be probably having a new variable, for example ANDROIDBUILD and when this variable exists and has whatever value, the build process does not start the ./os-features-test-netpbm_.

BR,

Ladislav

[dstndstn]

Okay -- You could also probably just create your own astrometry/include /os-features-config.h file containing the contents you want. As long as you 'touch' it, the Makefile shouldn't try to re-create it, right?

On Wed, Dec 8, 2021 at 3:59 PM Ladislav Heller @.***> wrote:

@dstndstn https://github.com/dstndstn

It's a dirty hack simply to get the things working on my system, but shorty I replaced the call of the binary ./os-features-test-netpbm (and redirection) here:

https://github.com/dstndstn/astrometry.net/blob/4b227b6b70befa6fa4fa7ee154d719c60236d65b/util/Makefile#L272

and here:

https://github.com/dstndstn/astrometry.net/blob/4b227b6b70befa6fa4fa7ee154d719c60236d65b/util/Makefile#L299

with echo ""

With the above hack the test was evaluated to True and HAVE_NETPBM was set to Yes.

A good solution to the above case would be probably having a new variable, for example ANDROID_BUILD and when this variable exists and has whatever value, the build process does not start the ./os-features-test-netpbm.

BR,

Ladislav

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/dstndstn/astrometry.net/issues/246#issuecomment-989190645, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAIEH7IJSZPOTYJTWEPCJT3UP7BL3ANCNFSM5JFVM4TQ . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

[laheller]

@dstndstn

Did not try that. I'll check. BTW what about the makefile.os-features file? This one is also generated on-the-fly and is somehow related...

[dstndstn]

Oh yeah, good point. makefile.os-features does a similar thing, changing Makefile behavior based on whether you have netpbm. FWIW, my makefile.os-features contains:

# This file is generated by util/Makefile.
HAVE_NETPBM := yes

and include/astrometry/os-features-config.h contains:

#define HAVE_NETPBM 1

On Wed, Dec 8, 2021 at 4:04 PM Ladislav Heller @.***> wrote:

Okay -- You could also probably just create your own astrometry/include /os-features-config.h file containing the contents you want. As long as you 'touch' it, the Makefile shouldn't try to re-create it, right? … <#m1450579198886327900> On Wed, Dec 8, 2021 at 3:59 PM Ladislav Heller @.**> wrote: @dstndstn https://github.com/dstndstn https://github.com/dstndstn It's a dirty hack simply to get the things working on my system, but shorty I replaced the call of the binary ./os-features-test-netpbm (and redirection) here: https://github.com/dstndstn/astrometry.net/blob/4b227b6b70befa6fa4fa7ee154d719c60236d65b/util/Makefile#L272 and here: https://github.com/dstndstn/astrometry.net/blob/4b227b6b70befa6fa4fa7ee154d719c60236d65b/util/Makefile#L299 with echo "" With the above hack the test was evaluated to True and HAVE_NETPBM was set to Yes. A good solution to the above case would be probably having a new variable, for example ANDROID_BUILD and when this variable exists and has whatever value, the build process does not start the ./os-features-test-netpbm*. BR, Ladislav — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#246 (comment) https://github.com/dstndstn/astrometry.net/issues/246#issuecomment-989190645>, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAIEH7IJSZPOTYJTWEPCJT3UP7BL3ANCNFSM5JFVM4TQ . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub .

Did not try that. I'll check. BTW what about the makefile.os-features file? This one is also generated on-the-fly and is somehow related...

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/dstndstn/astrometry.net/issues/246#issuecomment-989194370, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAIEH7LNRW4KLOA2Z32QBK3UP7B7RANCNFSM5JFVM4TQ . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

[laheller]

@dstndstn

OK I'll try to just create those two files. Anyway I think we can close this issue.

BR,

Ladislav

[readme9txt]

Hi @laheller I'm also recently trying to compile astrometry.net for the Android platform. But according to the documentation, source extraction of images requires python. Would you mind telling me how you solved this problem?

[dstndstn]

Did you read that you can use Source Extractor to do the source extraction?

[laheller]

Hi @laheller I'm also recently trying to compile astrometry.net for the Android platform. But according to the documentation, source extraction of images requires python. Would you tell me how you solved this problem?

Hi @readme9txt

Python is NOT required when you call astrometry.net like this (example): solve-field -O -L 0.1 -H 180.0 -u dw -z 2 -p -y -9 --uniformize 0 --fits-image ./YourAstroPhoto.fits

Important cmdline switches and params are: -p -9 --uniformize 0 --fits-image

Call "solve-field --help" for details about cmdline help.

Additionally the inuput format for asteometry.net (for solve-field) must be FITS!

When the above conditions are met, Python is completely skipped.

BR,

Ladislav

[readme9txt]

Did you read that you can use Source Extractor to do the source extraction?

Hi @dstndstn Yes i read it but i am not sure if it will work on android, i will try to compile it

[readme9txt]

@laheller It works! Thank you very mach for quick response!

[laheller]

@laheller It works! Thank you very mach for quick response!

@readme9txt BTW I was able to build astrometry.net for Android. I also created an application which calls it. You can try the early beta version: https://drive.google.com/file/d/1lb5A8OHhHyAivqD6opkFocMzj-8XE_t7/view

Demo video: https://youtu.be/wzrghuBEKa0

BR,

Ladislav

[readme9txt]

@laheller wow this is cool, it works fine on my Sony XZ2 (except I get some warning logs on first run, does that matter?)

[laheller]

@laheller wow this is cool, it works fine on my Sony XZ2 (except I get some warning logs on first run, does that matter?)

@readme9txt You should not see such warnings. Beside that did all the app functions work? Were you able to do plate solving?

[readme9txt]

@laheller Yes, everything works fine, so does plate solving (ps: my phone is not rooted)

[laheller]

@laheller Yes, everything works fine, so does plate solving (ps: my phone is not rooted)

@readme9txt Root is NOT required to make it work.

[readme9txt]

@laheller After a week of trying, I managed to build the dependencies of astrometry.net (cairo/netpbm/bzip2/cfitsio/gsl/wcslib) for AArch64 Android, but when I finish building and call make extra && make install I get some error logs

/usr/include/python3.8/pyconfig.h:9:12: fatal error: 'aarch64-linux-gnu/python3.8/pyconfig.h' file not found

It didn't interrupt the build process, after that I also got a bunch of programs in /usr/local/astrometry/bin So can I ignore these logs if I don't use python?

[laheller]

@laheller After a week of trying, I managed to build the dependencies of astrometry.net (cairo/netpbm/bzip2/cfitsio/gsl/wcslib) for AArch64 Android, but when I finish building and call make extra && make install I get some error logs

/usr/include/python3.8/pyconfig.h:9:12: fatal error: 'aarch64-linux-gnu/python3.8/pyconfig.h' file not found

It didn't interrupt the build process, after that I also got a bunch of programs in /usr/local/astrometry/bin So can I ignore these logs if I don't use python?

@readme9txt

You can ignore them.

Regarding the generated binaries, you probably will need only 4 of them:

• solve-field
• wcsinfo
• astrometry-engine
• plot-constellations (only if you want to tag the solved images)

Try to unzip the APK file I shared with you in the above comments and look into the Assets folder, where I put all the binaries needed by the app.