StellarSolver for Android?

laheller commented 2 years ago

Hi @rlancaste

This is not an issue, just a question.

What do you think, is it possible to build this library for Android platform using the Android NDK?

BR,

Ladislav

rlancaste commented 2 years ago

So that I do not know, I do not have any experience with that. A big first question is whether all the dependencies are supported on Android. If so, then it could be possible.

laheller commented 2 years ago

Hi @rlancaste

The library mentioned in this comment is NOT a dependency of astrometry.net but it has a detailed description, how to build a library using NDK for Android platform. Can be used as a starting point. I also will try start with dependencies one by one and build them.

rlancaste commented 2 years ago

Would the idea for this be to have it included in KStars for Android?

laheller commented 2 years ago

Yes. Or just simply to have a plate solver for Android.

Update: I was able to build following dependencies for Android:

gsl
cairo
cfitsio
wcslib
netpbm (not really needed for astrometry.net, but I just built it, even when it was a bit tricky)

laheller commented 2 years ago

@rlancaste

I have an Android build of astrometry.net that seems to be working at least partially. Now trying the pure binaries (including also the dependency libraries and some index files) directly on my Samsung Galaxy S20+ device (rooted). The plate solving fails somehow at the astrometry-engine binary.

Output from Android Terminal:

y2s:/data/local/tmp/astrometry.net # ./astrometry-engine -v ../Solve/Sas.axy Config file "./../etc/astrometry.cfg" doesn't exist. Using config file "./astrometry.cfg" Auto-indexing directory "/data/local/tmp/index_files" ... Skipping directory /data/local/tmp/index_files/. Skipping directory /data/local/tmp/index_files/.. Checking file "/data/local/tmp/index_files/index-4107.fits" Index name "/data/local/tmp/index_files/index-4107.fits" is readable, using as index filename Checking file "/data/local/tmp/index_files/index-4108.fits" Index name "/data/local/tmp/index_files/index-4108.fits" is readable, using as index filename Checking file "/data/local/tmp/index_files/index-4109.fits" Index name "/data/local/tmp/index_files/index-4109.fits" is readable, using as index filename Checking file "/data/local/tmp/index_files/index-4110.fits" Index name "/data/local/tmp/index_files/index-4110.fits" is readable, using as index filename Checking file "/data/local/tmp/index_files/index-4111.fits" Index name "/data/local/tmp/index_files/index-4111.fits" is readable, using as index filename Checking file "/data/local/tmp/index_files/index-4112.fits" Index name "/data/local/tmp/index_files/index-4112.fits" is readable, using as index filename Checking file "/data/local/tmp/index_files/index-4113.fits" Index name "/data/local/tmp/index_files/index-4113.fits" is readable, using as index filename Checking file "/data/local/tmp/index_files/index-4114.fits" Index name "/data/local/tmp/index_files/index-4114.fits" is readable, using as index filename Checking file "/data/local/tmp/index_files/index-4115.fits" Index name "/data/local/tmp/index_files/index-4115.fits" is readable, using as index filename Checking file "/data/local/tmp/index_files/index-4116.fits" Index name "/data/local/tmp/index_files/index-4116.fits" is readable, using as index filename Checking file "/data/local/tmp/index_files/index-4117.fits" Index name "/data/local/tmp/index_files/index-4117.fits" is readable, using as index filename Checking file "/data/local/tmp/index_files/index-4118.fits" Index name "/data/local/tmp/index_files/index-4118.fits" is readable, using as index filename Checking file "/data/local/tmp/index_files/index-4119.fits" Index name "/data/local/tmp/index_files/index-4119.fits" is readable, using as index filename Trying to add index "/data/local/tmp/index_files/index-4119.fits". Index name "/data/local/tmp/index_files/index-4119.fits" is readable, using as index filename Index name "/data/local/tmp/index_files/index-4119.fits" is readable, using as index filename Index name "/data/local/tmp/index_files/index-4119.fits" is readable, using as index filename Index scale: [1400, 2000] arcmin, [84000, 120000] arcsec Index has 1728 quads and 1080 stars Trying to add index "/data/local/tmp/index_files/index-4118.fits". Index name "/data/local/tmp/index_files/index-4118.fits" is readable, using as index filename Index name "/data/local/tmp/index_files/index-4118.fits" is readable, using as index filename Index name "/data/local/tmp/index_files/index-4118.fits" is readable, using as index filename Index scale: [1000, 1400] arcmin, [60000, 84000] arcsec Index has 3072 quads and 1920 stars Trying to add index "/data/local/tmp/index_files/index-4117.fits". Index name "/data/local/tmp/index_files/index-4117.fits" is readable, using as index filename Index name "/data/local/tmp/index_files/index-4117.fits" is readable, using as index filename Index name "/data/local/tmp/index_files/index-4117.fits" is readable, using as index filename Index scale: [680, 1000] arcmin, [40800, 60000] arcsec Index has 4800 quads and 3000 stars Trying to add index "/data/local/tmp/index_files/index-4116.fits". Index name "/data/local/tmp/index_files/index-4116.fits" is readable, using as index filename Index name "/data/local/tmp/index_files/index-4116.fits" is readable, using as index filename Index name "/data/local/tmp/index_files/index-4116.fits" is readable, using as index filename Index scale: [480, 680] arcmin, [28800, 40800] arcsec Index has 9408 quads and 5880 stars Trying to add index "/data/local/tmp/index_files/index-4115.fits". Index name "/data/local/tmp/index_files/index-4115.fits" is readable, using as index filename Index name "/data/local/tmp/index_files/index-4115.fits" is readable, using as index filename Index name "/data/local/tmp/index_files/index-4115.fits" is readable, using as index filename Index scale: [340, 480] arcmin, [20400, 28800] arcsec Index has 19200 quads and 12000 stars Trying to add index "/data/local/tmp/index_files/index-4114.fits". Index name "/data/local/tmp/index_files/index-4114.fits" is readable, using as index filename Index name "/data/local/tmp/index_files/index-4114.fits" is readable, using as index filename Index name "/data/local/tmp/index_files/index-4114.fits" is readable, using as index filename Index scale: [240, 340] arcmin, [14400, 20400] arcsec Index has 37632 quads and 23520 stars Trying to add index "/data/local/tmp/index_files/index-4113.fits". Index name "/data/local/tmp/index_files/index-4113.fits" is readable, using as index filename Index name "/data/local/tmp/index_files/index-4113.fits" is readable, using as index filename Index name "/data/local/tmp/index_files/index-4113.fits" is readable, using as index filename Index scale: [170, 240] arcmin, [10200, 14400] arcsec Index has 76800 quads and 48000 stars Trying to add index "/data/local/tmp/index_files/index-4112.fits". Index name "/data/local/tmp/index_files/index-4112.fits" is readable, using as index filename Index name "/data/local/tmp/index_files/index-4112.fits" is readable, using as index filename Index name "/data/local/tmp/index_files/index-4112.fits" is readable, using as index filename Index scale: [120, 170] arcmin, [7200, 10200] arcsec Index has 150528 quads and 94080 stars Trying to add index "/data/local/tmp/index_files/index-4111.fits". Index name "/data/local/tmp/index_files/index-4111.fits" is readable, using as index filename Index name "/data/local/tmp/index_files/index-4111.fits" is readable, using as index filename Index name "/data/local/tmp/index_files/index-4111.fits" is readable, using as index filename Index scale: [85, 120] arcmin, [5100, 7200] arcsec Index has 292032 quads and 182520 stars Trying to add index "/data/local/tmp/index_files/index-4110.fits". Index name "/data/local/tmp/index_files/index-4110.fits" is readable, using as index filename Index name "/data/local/tmp/index_files/index-4110.fits" is readable, using as index filename Index name "/data/local/tmp/index_files/index-4110.fits" is readable, using as index filename Index scale: [60, 85] arcmin, [3600, 5100] arcsec Index has 580800 quads and 362959 stars Trying to add index "/data/local/tmp/index_files/index-4109.fits". Index name "/data/local/tmp/index_files/index-4109.fits" is readable, using as index filename Index name "/data/local/tmp/index_files/index-4109.fits" is readable, using as index filename Index name "/data/local/tmp/index_files/index-4109.fits" is readable, using as index filename Index scale: [42, 60] arcmin, [2520, 3600] arcsec Index has 1168066 quads and 724260 stars Trying to add index "/data/local/tmp/index_files/index-4108.fits". Index name "/data/local/tmp/index_files/index-4108.fits" is readable, using as index filename Index name "/data/local/tmp/index_files/index-4108.fits" is readable, using as index filename Index name "/data/local/tmp/index_files/index-4108.fits" is readable, using as index filename Index scale: [30, 42] arcmin, [1800, 2520] arcsec Index has 2320889 quads and 1291539 stars Trying to add index "/data/local/tmp/index_files/index-4107.fits". Index name "/data/local/tmp/index_files/index-4107.fits" is readable, using as index filename Index name "/data/local/tmp/index_files/index-4107.fits" is readable, using as index filename Index name "/data/local/tmp/index_files/index-4107.fits" is readable, using as index filename Index scale: [22, 30] arcmin, [1320, 1800] arcsec Index has 4454259 quads and 1909957 stars Reading file "../Solve/Sas.axy"... Set odds ratio to solve to 1e+09 (log = 20.7233) Couldn't allocate memory for a bl node! Segmentation fault 134|y2s:/data/local/tmp/astrometry.net #

The above error message comes from the bl.c source file. Is there any option or switch to force astrometry to use less memory? Or I don't know, what to do with this error.

Anyway I will try to rebuild while experimenting with the build settings.

BR,

Ladislav

laheller commented 2 years ago

@rlancaste

FYI, I successfully built the whole astrometry.net including dependencies. Tested were following tools from suite:

solve-field
astrometry-engine
wcsinfo
plot-constellations

All of them worked perfectly!

Question is, what's next?

BR,

Ladislav

Screenshot_20211209-200015_Terminal Emulator.jpg

rlancaste commented 2 years ago

Ah that's cool. So next would be to see if we can build StellarSolver on Android or if we have to make any changes to make it work there. If it does work on Android, then we could include plate solving and Ekos on Android perhaps

rlancaste commented 2 years ago

So I just asked Jasem about it and he said that there isn't anybody currently working on or maintaining KStars for Android. While we could certainly get StellarSolver working for android, based on your success so far, the issue would be that next step.

laheller commented 2 years ago

@rlancaste

OK, I will try to build StellarSolver for Android (armv7a and aarch64 architectures) and let you know. BTW when I build it, I should get a shared library, something like libStellarSolver.so, right?

Android NDK has LLVM toolchain with clang compiler, it works pretty much the same way, as in Linux.

rlancaste commented 2 years ago

If you just build the library you should get libstellarsolver in the lib directory, as well as some include files in the StellarSolver subfolder of the include directory. If you also build the tester, then you should get StellarSolverTester in the bin directory

laheller commented 2 years ago

Hi @rlancaste

Trying to build StellarSolver for Android platform, and the cmake somehow can't find the gsl library, but it founds cftitsio.

Here is how I call cmake:

#! /bin/bash

# Set these variables to suit your needs
NDK_PATH=/home/laheller/android-ndk-r23b
TOOLCHAIN="clang"
ANDROID_VERSION="29"

export PKG_CONFIG_PATH=${PKG_CONFIG_PATH}:${HOME}/android-repos/gsl/_deploy_aarch64/lib/pkgconfig
export PKG_CONFIG_PATH=${PKG_CONFIG_PATH}:${HOME}/android-repos/cfitsio-4.0.0/_deploy_aarch64/lib/pkgconfig
export PKG_CONFIG_PATH=${PKG_CONFIG_PATH}:${HOME}/android-repos/wcslib-7.7/_deploy_aarch64/lib/pkgconfig

export CFLAGS="-fPIC -std=c17"

cmake -G"Unix Makefiles" \
    -DANDROID_ABI=arm64-v8a \
    -DANDROID_PLATFORM=android-${ANDROID_VERSION} \
    -DANDROID_TOOLCHAIN=${TOOLCHAIN} \
    -DCMAKE_ASM_FLAGS="--target=aarch64-linux-android${ANDROID_VERSION}" \
    -DCMAKE_TOOLCHAIN_FILE=${NDK_PATH}/build/cmake/android.toolchain.cmake \
    -DCMAKE_INSTALL_PREFIX=$(pwd)/../_deploy_aarch64 \
    ..

...and here is the output:

-- Android: Targeting API '29' with architecture 'arm64', ABI 'arm64-v8a', and processor 'aarch64' -- Android: Selected unified Clang toolchain -- The C compiler identification is Clang 12.0.8 -- The CXX compiler identification is Clang 12.0.8 -- Detecting C compiler ABI info -- Detecting C compiler ABI info - done -- Check for working C compiler: /home/laheller/android-ndk-r23b/toolchains/llvm/prebuilt/linux-x86_64/bin/clang - skipped -- Detecting C compile features -- Detecting C compile features - done -- Detecting CXX compiler ABI info -- Detecting CXX compiler ABI info - done -- Check for working CXX compiler: /home/laheller/android-ndk-r23b/toolchains/llvm/prebuilt/linux-x86_64/bin/clang++ - skipped -- Detecting CXX compile features -- Detecting CXX compile features - done CMake Error: TRY_RUN() invoked in cross-compiling mode, please set the following cache variables appropriately: RUN_RESULT_2 (advanced) For details see /home/laheller/android-repos/stellarsolver/_build/TryRunResults.cmake -- Found PkgConfig: /usr/bin/pkg-config (found version "0.29.1")
-- Checking for module 'cfitsio' -- Found cfitsio, version 4.0.0 CMake Error at /usr/share/cmake-3.22/Modules/FindPackageHandleStandardArgs.cmake:230 (message): Could NOT find GSL (missing: GSL_INCLUDE_DIR GSL_LIBRARY GSL_CBLAS_LIBRARY) (found version "2.7+") Call Stack (most recent call first): /usr/share/cmake-3.22/Modules/FindPackageHandleStandardArgs.cmake:594 (_FPHSA_FAILURE_MESSAGE) cmake/modules/FindGSL.cmake:158 (find_package_handle_standard_args) CMakeLists.txt:93 (find_package)

-- Configuring incomplete, errors occurred! See also "/home/laheller/android-repos/stellarsolver/_build/CMakeFiles/CMakeOutput.log".

As you can see I am exporting PKG_CONFIG_PATH variable and setting its value find all the dependencies.

Any idea, why it does not work for GSL ?

BR,

Ladislav

rlancaste commented 2 years ago

I don't know if this is relevant or not, but sometimes when people have trouble finding GSL on their systems, it's because they don't have the dev version of GSL installed. Is that an option on Android?

laheller commented 2 years ago

@rlancaste

I have this version: https://www.gnu.org/software/gsl/#development

I built this for Android and as output I have headers and library files as usual.

BTW there is a note on the above page, that the gsl build should be configured with:

./configure --enable-maintainer-mode

Is that maybe required? I did not start configure with the above switch.

rlancaste commented 2 years ago

Were you able to get this to work on Android?

laheller commented 2 years ago

Hi @rlancaste

I was not able to build this project for Android. It always failed to detect GSL library. As I wrote earlier, I only built the original astrometry.net suite.

BR,

Ladislav

laheller commented 2 years ago

Hi @rlancaste

Not directly related to this topic, rather a general question to StellarSolver: How do you perform the parallel solving? Do you run more than one instances of solve-field with given parameters but each instance with their own backend config file? When yes, the difference between each backend config files are only the named index files?

BTW I gave up fighting with StellarSolver build for Android. The cmake simply cannot found the GSL dependency.

BR,

Ladislav

rlancaste commented 2 years ago

Hi there, it depends on what solver you are using. If you are using the external solver, it makes parallel threads and each one calls solve-field with different parameters. All the threads share the same config files. If you are using the internal solver, it creates parallel threads and each one runs the solving methods with different options. It doesn't use any config files at all in this case.

Sorry to hear about the issue with Android. Sorry I couldn't help, I have no experience with android.

laheller commented 2 years ago

Hi @rlancaste

In case of an external solver (solve-field) what is the benefit of parallel threads sharing the same config file? I would expect parallelism to speed up the solving time but I can't really imagine, how could the same shared config file help with that. In such a case for each thread, the same index files are processed in the same order. I am talking about the case of varying field input images. For example, for a narrow-field ( < 1 degree ) input image, the solver engine must process many index-files while finding the matching one. When all threads do exactly the same index files processing in the same order, it is just a waste of CPU/IO resources, or am I wrong? I am asking because currently, I am testing my Android build (of the original astrometry.net suite) and its "parent" application with only ONE solver thread. The thread simply calls solve-field and takes parameters from the app GUI. It is relatively fast for wide-field input images but apparently slows down on narrower fields. To be honest, I took the idea to speed up the things from your StellarSolver implementation, but I am not sure, how to directly do it on my side. My stupid idea is to group the index files by field width, create a config file for each group and finally start a dedicated solver thread for each config file. Not tried yet, I am just thinking loudly.

Regarding the StellarSolver build for Android: The problem is NOT related to Android. The build system is very similar to or almost the same as on the desktop Linux systems, it is an LLVM toolchain. Also, the build itself happens on an Ubuntu Linux machine. The problem is somewhere at cmake and/or CmakeLists.txt and/or modules/FindGSL.cmake.

BR,

Ladislav

rlancaste commented 2 years ago

The reason is because the config file just sets a couple of the parameters, most of the settings for astrometry.net get set using options, not the config file. In the separate parallel threads, to speed up the solve, I have multiple instances of astrometry searching different scales or alternately different "depths" (a term astrometry.net uses). You could also search different areas of the sky simultaneously, but I think there was a reason I didn't do that one. But I guess you could use the config file if you wanted to since there are options in there that could be set differently for different simultaneous solves. I don't want to do that myself because that means more files need to be generated and Raspberry Pis don't like too much of that. It was just as easy to parallelize the scales and depths, which are just different parameters/options and don't require extra files.

laheller commented 2 years ago

@rlancaste

OK, thanks for explanation. First I will try multiple solver threads with different scales option while sharing the same config.

BTW you can try my experimental Android build of astrometry.net if you want. I can send you the APK file you can directly install on your smartphone. Are you interested?

laheller commented 1 year ago

Hi @rlancaste

Trying to build StellarSolver for Android platform, and the cmake somehow can't find the gsl library, but it founds cftitsio.

Here is how I call cmake:
#! /bin/bash

# Set these variables to suit your needs
NDK_PATH=/home/laheller/android-ndk-r23b
TOOLCHAIN="clang"
ANDROID_VERSION="29"

export PKG_CONFIG_PATH=${PKG_CONFIG_PATH}:${HOME}/android-repos/gsl/_deploy_aarch64/lib/pkgconfig
export PKG_CONFIG_PATH=${PKG_CONFIG_PATH}:${HOME}/android-repos/cfitsio-4.0.0/_deploy_aarch64/lib/pkgconfig
export PKG_CONFIG_PATH=${PKG_CONFIG_PATH}:${HOME}/android-repos/wcslib-7.7/_deploy_aarch64/lib/pkgconfig

export CFLAGS="-fPIC -std=c17"

cmake -G"Unix Makefiles" \
    -DANDROID_ABI=arm64-v8a \
    -DANDROID_PLATFORM=android-${ANDROID_VERSION} \
    -DANDROID_TOOLCHAIN=${TOOLCHAIN} \
    -DCMAKE_ASM_FLAGS="--target=aarch64-linux-android${ANDROID_VERSION}" \
    -DCMAKE_TOOLCHAIN_FILE=${NDK_PATH}/build/cmake/android.toolchain.cmake \
    -DCMAKE_INSTALL_PREFIX=$(pwd)/../_deploy_aarch64 \
    ..
...and here is the output:

-- Android: Targeting API '29' with architecture 'arm64', ABI 'arm64-v8a', and processor 'aarch64' -- Android: Selected unified Clang toolchain -- The C compiler identification is Clang 12.0.8 -- The CXX compiler identification is Clang 12.0.8 -- Detecting C compiler ABI info -- Detecting C compiler ABI info - done -- Check for working C compiler: /home/laheller/android-ndk-r23b/toolchains/llvm/prebuilt/linux-x86_64/bin/clang - skipped -- Detecting C compile features -- Detecting C compile features - done -- Detecting CXX compiler ABI info -- Detecting CXX compiler ABI info - done -- Check for working CXX compiler: /home/laheller/android-ndk-r23b/toolchains/llvm/prebuilt/linux-x86_64/bin/clang++ - skipped -- Detecting CXX compile features -- Detecting CXX compile features - done CMake Error: TRY_RUN() invoked in cross-compiling mode, please set the following cache variables appropriately: RUN_RESULT_2 (advanced) For details see /home/laheller/android-repos/stellarsolver/_build/TryRunResults.cmake -- Found PkgConfig: /usr/bin/pkg-config (found version "0.29.1") -- Checking for module 'cfitsio' -- Found cfitsio, version 4.0.0 CMake Error at /usr/share/cmake-3.22/Modules/FindPackageHandleStandardArgs.cmake:230 (message): Could NOT find GSL (missing: GSL_INCLUDE_DIR GSL_LIBRARY GSL_CBLAS_LIBRARY) (found version "2.7+") Call Stack (most recent call first): /usr/share/cmake-3.22/Modules/FindPackageHandleStandardArgs.cmake:594 (_FPHSA_FAILURE_MESSAGE) cmake/modules/FindGSL.cmake:158 (find_package_handle_standard_args) CMakeLists.txt:93 (find_package) -- Configuring incomplete, errors occurred! See also "/home/laheller/android-repos/stellarsolver/_build/CMakeFiles/CMakeOutput.log".

As you can see I am exporting PKG_CONFIG_PATH variable and setting its value find all the dependencies.

Any idea, why it does not work for GSL ?

BR,

Ladislav

Hi @rlancaste

A bit late but back to the quoted question: I finally managed to find the GSL library, cmake is now able to detect it.

Now the additional question: Is it possible to build ONLY the library (.so files) and SKIP all the GUI stuff (with QT5 dependency) completely?

BR,

Ladislav

rlancaste commented 1 year ago

Hi again Ladislav,

Good to hear that you have progress! As for skipping QT5, I don't think that would be possible right now because I used QT a lot in the coding of the main classes for StellarSolver. Yes, you can certainly build the library without the GUI, since that is done all the time for KStars on all the OS platforms, but it still relies on QT classes. Certainly if you find a way to avoid that, or make a change that might support everything it currently does but does not make other things not work or sacrifice its cross platform performance, then wonderful and I would be glad to accept changes. At the moment, I can't really embark on a big project like that because I am in the middle of starting up the school year since I am a teacher, and I am also taking classes for a masters degree, and working on some other projects as well. But if you want to try to make changes and want me to check them out and provide feedback, or accept changes once you add something, then great!

Thanks,

Rob

laheller commented 1 year ago

@rlancaste

OK, understand. Just one more question: In the StellarSolver library itself are any GUI-related QT5 function calls? If not, I maybe can workaround/eliminate them.

rlancaste commented 1 year ago

No, the library itself is just meant to be a workhorse for programs like kstars. As far as I recall, the GUI related items are all in the Tester program, the batch solver program, the utils library (for other programs to use), and other example programs.

laheller commented 1 year ago

No, the library itself is just meant to be a workhorse for programs like kstars. As far as I recall, the GUI related items are all in the Tester program, the batch solver program, the utils library (for other programs to use), and other example programs.

@rlancaste

I mean for example the Parameters.h includes QObject

and similar things. I guess they are some basic QT5 data types and nothing GUI related...

rlancaste commented 1 year ago

Yes, I used QT as the basis for my coding since it is cross platform and KStars is a QT program. That stuff should not be related to GUI since the stellarsolver library has no GUI. The other programs based on it do, but the library itself does not.

rlancaste commented 1 year ago

if we make big changes like in parameters.h or data types though, we will need to take our time and do it in a branch and get feedback from others and test it thoroughly, since changing data types that the other programs are built upon will change what they are expecting to get. Even then, making changes like that can cause problems still because of people using older versions of the library vs. newer versions of kstars or vice versa.

rlancaste commented 1 year ago

I know I changed the data structs a few years ago, and it was a few weeks to months before all the issues worked themselves out

laheller commented 1 year ago

I know I changed the data structs a few years ago, and it was a few weeks to months before all the issues worked themselves out

@rlancaste Sure, nothing into main branch. This is not even my plan. First of all I would like to make at least the library work on Android. Build it locally, etc. After that we can talk about a separate branch maybe.

rlancaste commented 1 year ago

Sounds great!

laheller commented 1 year ago

Hi @rlancaste

Now I am at the point when the linking fails - output of make command:

[100%] Linking CXX shared library android-build/libs/arm64-v8a/libstellarsolver_arm64-v8a.so ld: error: undefined symbol: qsort_r referenced by bl-sort.c:68 (/home/laheller/repos/stellarsolver/stellarsolver/astrometry/util/bl-sort.c:68) CMakeFiles/stellarsolver.dir/stellarsolver/astrometry/util/bl-sort.c.o:(bl_sort_rec) referenced by permutedsort.c:86 (/home/laheller/repos/stellarsolver/stellarsolver/astrometry/util/permutedsort.c:86) CMakeFiles/stellarsolver.dir/stellarsolver/astrometry/util/permutedsort.c.o:(permuted_sort) clang-12: error: linker command failed with exit code 1 (use -v to see invocation) make[2]: [CMakeFiles/stellarsolver.dir/build.make:1238: android-build/libs/arm64-v8a/libstellarsolver_arm64-v8a.so] Error 1 make[1]: [CMakeFiles/Makefile2:139: CMakeFiles/stellarsolver.dir/all] Error 2 make: *** [Makefile:136: all] Error 2

Any idea, what is this qsort_r symbol, what library it is coming from or how to workaround it? Interesting that as I mentioned in the above comments, I was able to build the full original astrometry.net suite from sources for Android and never had such linking error.

BR,

Ladislav

rlancaste commented 1 year ago

So, qsort-r is a system method and it as well as several other features are tested for by astrometry.net in os-features.c which is one of the first things that cmake does on your system. As far as I remember, it runs a test program and if it finds the symbol in your system, it will use that method, but if not, it will declare its own method to perform the function. That way there won't be a conflict with two symbols with the same name. I found it to be a little clunky, but I tried my best to implement that functionality in stellarsolver as close to what happens in astrometry.net as I could. When I made stellarsolver in 2020, it was on version 0.76. I think the fixed this issue in version 0.82 by removing those tests and always using their in-house functions somehow. Updating the code now so that it is based on the latest astrometry code would take a lot of work and I cannot undertake that project now. But during a summer I certainly could.

To see what is wrong in your build, why don't you check what happened after the test code was run. The results are saved in the include folder in os-features-config.h. Perhaps on your system qsort-r needed to be declared and set up but it was not?

laheller commented 1 year ago

@rlancaste

Good news! Finally, I built StellarSolver library plus tests and demos for Android. For the qsort_r issue, it was enough to set NEED_QSORT_R to "1" in os-features-config.h before calling make. For each of the tests and demos sources, I had to include header to build them successfully. Besides the above, I had to download and install QT for Android, I used Qt 5.14.2 for Linux hosts.

Now the question is, what do we want to do with this? Maybe a PR to a new branch? So far my only idea is to create a demo Android application which using the StellarSolver library can platesolve an input FITS file.

Update: I realized that the whole StellarSolver library was written in C++ language. But since I plan to use it from an Android application, I think we need to create a small C-language wrapper around the library with "extern" functions and access those functions from Android app. This is very common when we want to access library functions from languages other than C++.

Example: Before actual plate solving we have to call fileio/loadImage. For this we need the below C-wrapper functions:

fileio_create (creates and returns a pointer to instance of fileio class)
fileio_destroy (destroys the instance of fileio class specified by the pointer created above)
fileio_loadImage (using the pointer to instance and filename does what the name suggests)

Sarajevo67 commented 6 months ago

Hi guys,

first I would like to congratulate both of you, for the huge effort invested into porting a very complex code.

I am currently dealing with the acquisition of burst stack astrophotography, using the Android camera2 API. You can imagine with how much delight I discovered your work on SEP and solver customization.

My ambition is to forward a stacked pan uint32 little-endian array to Android SEP (without FITS conversion), for background and source extraction. A less ambitious goal, involves intermediate DNG to FITS step, but I would like to avoid it...

My kind questions, for you: @laheller What is current status of your port? Is it available as fork? @rlancaste Is there any chance to merge this Android port?

With Best Regards,

DD

rlancaste commented 6 months ago

Hi Sarajevo67,

Sure I can definitely include this if it can be included in a way that doesn't break anything else needed for other operating systems. A number of big projects like KStars currently depend heavily on this and I don't want to break it. laheller, I would be glad to include your work if you can get together a pull request and we test it throughly to make sure it doesn't affect other things. We could also add scripts and instructions for working with it on Android vs. the other systems. Sorry I have been kind of busy recently and haven't worked on much, but I am going to have more time in the spring/summer that I can work on things again.

Thanks,

Rob

laheller commented 6 months ago

Hi @Sarajevo67 @rlancaste

From my side what I miss in this library is a small C language wrapper around some of the C++ API functions, see my previous comment for examples. Beside that I already built the library for Android and I have the binaries. I can share the build sripts.

BTW what is your goal with Android Camera2? To build a camera app which directly plate solves the photos taken?

BR,

Ladislav

Sarajevo67 commented 6 months ago

@laheller

From my side what I miss in this library is a small C language wrapper around some of the C++ API functions, see my previous comment for examples. Beside that I already built the library for Android and I have the binaries. I can share the build sripts.

It is library anyway and adaptation for different job entry points looks natural. For me it is around bayer.c, but it depends what @rlancaste find appropriate, that doesn't break anything else needed for other operating systems. Please share your build scripts.

BTW what is your goal with Android Camera2? To build a camera app which directly plate solves the photos taken?

Yes, and it is, most probably, true intent of Android port. Originally I wanted to use Mark Harman Open Camera TakePhoto widget, but overall package is too massive for simple, single purpose infinity/ISO/exposure/burstCount - acquisition. When we have embedded acquisition, it seems logical to do Bayer level stacking, noise and hot pixel removal in NDK, then forward array to bayer.c, for demosaicing (in that case single uint16 array is forwarded). Your suggestions are welcome.

laheller commented 6 months ago

@Sarajevo67

OK, I will share my build scripts including some instructions in the fork of this repo. Once available, will let you know.

A hint to the raw images processing got from Camera2 API: Consider using the OpenCV library for that, since it is one of the bests or maybe THE best for such a purpose. I also built this library for Android.

laheller commented 6 months ago

@Sarajevo67

You can clone the fork here and follow the instructions: https://github.com/laheller/stellarsolver/blob/master/_build/README.Android

Sarajevo67 commented 6 months ago

Thank you!!!

BR,

DD

Sarajevo67 commented 6 months ago

Hi @laheller

Consider using the OpenCV library for that, since it is one of the bests or maybe THE best for such a purpose.

I couldn't find in OpenCV (in SEP also), pre demosaicing, hot pixel detection/correction? Most of tools available, doing hot pix processing after deBayer, when noise already blended in the neighborhood. Stellar Solver utils bayer.c, has nice demosaicing collection, from simple to HQ and it would be great to complement it with hot pixel correction. Problem is not a trivial one and traditional 'difference from median' approach, does not work well for automated astrophotography (2 pixel distance for same color R and B pixels). Have you seen anything applicable for this issue?

BR,

DD

laheller commented 6 months ago

Hi @laheller

Consider using the OpenCV library for that, since it is one of the bests or maybe THE best for such a purpose.

I couldn't find in OpenCV (in SEP also), pre demosaicing, hot pixel detection/correction? Most of tools available, doing hot pix processing after deBayer, when noise already blended in the neighborhood. Stellar Solver utils bayer.c, has nice demosaicing collection, from simple to HQ and it would be great to complement it with hot pixel correction. Problem is not a trivial one and traditional 'difference from median' approach, does not work well for automated astrophotography (2 pixel distance for same color R and B pixels). Have you seen anything applicable for this issue?

BR,

DD

@Sarajevo67 Demosaicing is part of OpenCV imgproc module:

The hot pixel detection/correction could be done by using OpenCV's median filter function, as described in this answer.

Good Luck!

taurushappy commented 5 months ago

boofcv( java image deal lib) can be instead of opencv to deal image

taurushappy commented 5 months ago

https://github.com/lessthanoptimal/BoofCV
https://boofcv.org/index.php?title=Main_Page

taurushappy commented 3 months ago

Hi @rlancaste

Trying to build StellarSolver for Android platform, and the cmake somehow can't find the gsl library, but it founds cftitsio.

Here is how I call cmake:
#! /bin/bash

# Set these variables to suit your needs
NDK_PATH=/home/laheller/android-ndk-r23b
TOOLCHAIN="clang"
ANDROID_VERSION="29"

export PKG_CONFIG_PATH=${PKG_CONFIG_PATH}:${HOME}/android-repos/gsl/_deploy_aarch64/lib/pkgconfig
export PKG_CONFIG_PATH=${PKG_CONFIG_PATH}:${HOME}/android-repos/cfitsio-4.0.0/_deploy_aarch64/lib/pkgconfig
export PKG_CONFIG_PATH=${PKG_CONFIG_PATH}:${HOME}/android-repos/wcslib-7.7/_deploy_aarch64/lib/pkgconfig

export CFLAGS="-fPIC -std=c17"

cmake -G"Unix Makefiles" \
    -DANDROID_ABI=arm64-v8a \
    -DANDROID_PLATFORM=android-${ANDROID_VERSION} \
    -DANDROID_TOOLCHAIN=${TOOLCHAIN} \
    -DCMAKE_ASM_FLAGS="--target=aarch64-linux-android${ANDROID_VERSION}" \
    -DCMAKE_TOOLCHAIN_FILE=${NDK_PATH}/build/cmake/android.toolchain.cmake \
    -DCMAKE_INSTALL_PREFIX=$(pwd)/../_deploy_aarch64 \
    ..
...and here is the output:

-- Android: Targeting API '29' with architecture 'arm64', ABI 'arm64-v8a', and processor 'aarch64' -- Android: Selected unified Clang toolchain -- The C compiler identification is Clang 12.0.8 -- The CXX compiler identification is Clang 12.0.8 -- Detecting C compiler ABI info -- Detecting C compiler ABI info - done -- Check for working C compiler: /home/laheller/android-ndk-r23b/toolchains/llvm/prebuilt/linux-x86_64/bin/clang - skipped -- Detecting C compile features -- Detecting C compile features - done -- Detecting CXX compiler ABI info -- Detecting CXX compiler ABI info - done -- Check for working CXX compiler: /home/laheller/android-ndk-r23b/toolchains/llvm/prebuilt/linux-x86_64/bin/clang++ - skipped -- Detecting CXX compile features -- Detecting CXX compile features - done CMake Error: TRY_RUN() invoked in cross-compiling mode, please set the following cache variables appropriately: RUN_RESULT_2 (advanced) For details see /home/laheller/android-repos/stellarsolver/_build/TryRunResults.cmake -- Found PkgConfig: /usr/bin/pkg-config (found version "0.29.1") -- Checking for module 'cfitsio' -- Found cfitsio, version 4.0.0 CMake Error at /usr/share/cmake-3.22/Modules/FindPackageHandleStandardArgs.cmake:230 (message): Could NOT find GSL (missing: GSL_INCLUDE_DIR GSL_LIBRARY GSL_CBLAS_LIBRARY) (found version "2.7+") Call Stack (most recent call first): /usr/share/cmake-3.22/Modules/FindPackageHandleStandardArgs.cmake:594 (_FPHSA_FAILURE_MESSAGE) cmake/modules/FindGSL.cmake:158 (find_package_handle_standard_args) CMakeLists.txt:93 (find_package) -- Configuring incomplete, errors occurred! See also "/home/laheller/android-repos/stellarsolver/_build/CMakeFiles/CMakeOutput.log".

As you can see I am exporting PKG_CONFIG_PATH variable and setting its value find all the dependencies.

Any idea, why it does not work for GSL ?

BR,

Ladislav

sir, I encouter the same problem , Could NOT find GSL (miss :GSL_INCLUDE_DIR) , Can U share how to solver the problem,

Hi @rlancaste

Trying to build StellarSolver for Android platform, and the cmake somehow can't find the gsl library, but it founds cftitsio.

Here is how I call cmake:
#! /bin/bash

# Set these variables to suit your needs
NDK_PATH=/home/laheller/android-ndk-r23b
TOOLCHAIN="clang"
ANDROID_VERSION="29"

export PKG_CONFIG_PATH=${PKG_CONFIG_PATH}:${HOME}/android-repos/gsl/_deploy_aarch64/lib/pkgconfig
export PKG_CONFIG_PATH=${PKG_CONFIG_PATH}:${HOME}/android-repos/cfitsio-4.0.0/_deploy_aarch64/lib/pkgconfig
export PKG_CONFIG_PATH=${PKG_CONFIG_PATH}:${HOME}/android-repos/wcslib-7.7/_deploy_aarch64/lib/pkgconfig

export CFLAGS="-fPIC -std=c17"

cmake -G"Unix Makefiles" \
    -DANDROID_ABI=arm64-v8a \
    -DANDROID_PLATFORM=android-${ANDROID_VERSION} \
    -DANDROID_TOOLCHAIN=${TOOLCHAIN} \
    -DCMAKE_ASM_FLAGS="--target=aarch64-linux-android${ANDROID_VERSION}" \
    -DCMAKE_TOOLCHAIN_FILE=${NDK_PATH}/build/cmake/android.toolchain.cmake \
    -DCMAKE_INSTALL_PREFIX=$(pwd)/../_deploy_aarch64 \
    ..
...and here is the output:

-- Android: Targeting API '29' with architecture 'arm64', ABI 'arm64-v8a', and processor 'aarch64' -- Android: Selected unified Clang toolchain -- The C compiler identification is Clang 12.0.8 -- The CXX compiler identification is Clang 12.0.8 -- Detecting C compiler ABI info -- Detecting C compiler ABI info - done -- Check for working C compiler: /home/laheller/android-ndk-r23b/toolchains/llvm/prebuilt/linux-x86_64/bin/clang - skipped -- Detecting C compile features -- Detecting C compile features - done -- Detecting CXX compiler ABI info -- Detecting CXX compiler ABI info - done -- Check for working CXX compiler: /home/laheller/android-ndk-r23b/toolchains/llvm/prebuilt/linux-x86_64/bin/clang++ - skipped -- Detecting CXX compile features -- Detecting CXX compile features - done CMake Error: TRY_RUN() invoked in cross-compiling mode, please set the following cache variables appropriately: RUN_RESULT_2 (advanced) For details see /home/laheller/android-repos/stellarsolver/_build/TryRunResults.cmake -- Found PkgConfig: /usr/bin/pkg-config (found version "0.29.1") -- Checking for module 'cfitsio' -- Found cfitsio, version 4.0.0 CMake Error at /usr/share/cmake-3.22/Modules/FindPackageHandleStandardArgs.cmake:230 (message): Could NOT find GSL (missing: GSL_INCLUDE_DIR GSL_LIBRARY GSL_CBLAS_LIBRARY) (found version "2.7+") Call Stack (most recent call first): /usr/share/cmake-3.22/Modules/FindPackageHandleStandardArgs.cmake:594 (_FPHSA_FAILURE_MESSAGE) cmake/modules/FindGSL.cmake:158 (find_package_handle_standard_args) CMakeLists.txt:93 (find_package) -- Configuring incomplete, errors occurred! See also "/home/laheller/android-repos/stellarsolver/_build/CMakeFiles/CMakeOutput.log".

As you can see I am exporting PKG_CONFIG_PATH variable and setting its value find all the dependencies.

Any idea, why it does not work for GSL ?

BR,

Ladislav

sir, I encouter the same problem , Could NOT find GSL (miss :GSL_INCLUDE_DIR) , Can U share how to solver the problem,

laheller commented 3 months ago

sir, I encouter the same problem , Could NOT find GSL (miss :GSL_INCLUDE_DIR) , Can U share how to solver the problem,

It won't work using the export PKG_CONFIG_PATH=...

You have to specify in cmake command line: -DGSL_INCLUDE_DIRS=/path/to/the/folder/where/gsl/include/files/are/located -DGSL_LIBRARIES=/path/to/the/file/libgsl.so:/path/to/the/file/libgsl-cblas.so

taurushappy commented 2 months ago

sir, I encouter the same problem , Could NOT find GSL (miss :GSL_INCLUDE_DIR) , Can U share how to solver the problem,

It won't work using the export PKG_CONFIG_PATH=...

You have to specify in cmake command line: -DGSL_INCLUDE_DIRS=/path/to/the/folder/where/gsl/include/files/are/located -DGSL_LIBRARIES=/path/to/the/file/libgsl.so:/path/to/the/file/libgsl-cblas.so

thank u sir ! BR

rlancaste / stellarsolver

StellarSolver for Android? #85