The prebuilt Android command line version is actually not an executable but a library

[laheller]

Hi @han-k59

The prebuilt CLI versions for Android (aarch64 and armhf) available here are actually NOT built as executable apps but a shared objects (libraries).

When I run the Linux readelf tool on them, to display the header:

readelf -h /data/local/tmp/astap/astap_cli

the output is following:

ELF Header: Magic: 7f 45 4c 46 02 01 01 00 00 00 00 00 00 00 00 00 Class: ELF64 Data: 2's complement, little endian Version: 1 (current) OS/ABI: UNIX - System V ABI Version: 0 Type: DYN (Shared object file) Machine: arm64 Version: 0x1 Entry point address: 0x289e0 Start of program headers: 64 (bytes into file) Start of section headers: 972208 (bytes into file) Flags: 0x0 Size of this header: 64 (bytes) Size of program headers: 56 (bytes) Number of program headers: 8 Size of section headers: 64 (bytes) Number of section headers: 24 Section header string table index: 23

Which is not correct, because the Type in case of executables should be "EXEC (Executable file)". It means that the downloadable astap_cli is not built properly, or at least not intended to do what its name (CLI) suggests.

BR,

Ladislav

[han-k59]

Thanks for the feedback. Looks like something went wrong in the last versions. The older version is executable:

Latest version: h@h-vmwarevirtualplatform:~$ file astap_cli astap_cli: ELF 64-bit LSB shared object, ARM aarch64, version 1 (SYSV), dynamically linked, interpreter libdl.so, BuildID[sha1]=bb58f96998c6d55628402334f44e928c1cbd56cc, stripped

Older version from: https://sourceforge.net/projects/astap-program/files/linux_installer/older/ h@h-vmwarevirtualplatform:~$ file astap_cli astap_cli: ELF 64-bit LSB executable, ARM aarch64, version 1 (SYSV), statically linked, stripped h@h-vmwarevirtualplatform:~$

I will fix it. You could use the older version for the time being.

Han

[han-k59]

I have tested the application in Bluestacks emulator and it executes normal. The question is now what to blame. The application or the reporting of readelf?

[laheller]

I have tested the application in Bluestacks emulator and it executes normal. The question is now what to blame. The application or the reporting of readelf?

@han-k59

I don't know what to blame, but when I try to execute the current latest version (both aarch64 and arm) from command line, it displays only the below message:

a54x:/data/local/tmp/astap # ls -la total 958 drwxr-xr-x 2 root root 3452 2023-12-15 20:30 . drwxrwx--x 3 shell shell 3452 2023-12-15 20:28 .. -rwxr-xr-x 1 root root 973744 2023-12-15 20:30 astap_cli a54x:/data/local/tmp/astap # a54x:/data/local/tmp/astap # ./astap_cli /system/bin/sh: ./astap_cli: No such file or directory

As you can see, the astap_cli file is there in the folder and also has executable permissions. The "No such file or directory" message might be misleading, but for sure happens in case, when the file is actually NOT an executable.

Anyway, going to check the previous version.

[laheller]

@han-k59

The version 2022-04-23 works as expected:

a54x:/data/local/tmp/astap # ./astap_cli ASTAP astrometric solver, CLI-2022-4-10 (C) 2018, 2022 by Han Kleijn. License MPL 2.0, Webpage: www.hnsky.org Usage: -f filename {fits, tiff, png, jpg files} -r radius_area_to_search[degrees] -z downsample_factor[0,1,2,3,4] {Downsample prior to solving. 0 is auto} -fov diameter_field[degrees] -ra center_right ascension[hours] -spd center_south_pole_distance[degrees] -s max_number_of_stars {default 500, 0 is auto} -t tolerance {default 0.007} -m minimum_star_size["] {default 1.5} -check apply[y/n] {Apply check pattern filter prior to solving. Use for raw OSC images only when binning is 1x1} -speed mode[auto/slow] {Slow is forcing reading a larger area from the star database (more overlap) to improve detection} -o file {Name the output files with this base path & file name} -d path {specify a path to the star database} -analyse snr_min {Analyse only and report median HFD and number of stars used} -extract snr_min {As -analyse but additionally write a .csv file with the detected stars info} -log {Write the solver log to file} -progress {Log all progress steps and messages} -update {update the FITS header with the found solution. Jpg, png, tiff will be written as fits} -wcs {Write a .wcs file in similar format as Astrometry.net. Else text style.} Preference will be given to the command line values.

...and the readelf tool says:

a54x:/data/local/tmp/astap # readelf -h ./astap_cli ELF Header: Magic: 7f 45 4c 46 02 01 01 00 00 00 00 00 00 00 00 00 Class: ELF64 Data: 2's complement, little endian Version: 1 (current) OS/ABI: UNIX - System V ABI Version: 0 Type: EXEC (Executable file) Machine: arm64

[han-k59]

Thanks for the feedback. What you observe is worrying. I will do the following:

1) Report this at bug tracker of the FPC compiler. 2) Contact the maker of OpenLiverstacker who is using ASTAP_CLI for his Android app. https://www.cloudynights.com/topic/872895-openlivestacker-4th-beta-android-stability-fixes/ 3) Try to setup again the old compiler allowing to compile the latest version.

Han

[han-k59]

I have found the version from 2023-05-14 and placed them in this folder:

https://sourceforge.net/projects/astap-program/files/android/older/

They are probably the last good one. Strange enough, I can't compile again. Looks like it is in the operating system upgrade??. I will try in an older Linux version.

[artyom-beilis]

It is same for one I run

ELF Header:
  Magic:   7f 45 4c 46 02 01 01 00 00 00 00 00 00 00 00 00 
  Class:                             ELF64
  Data:                              2's complement, little endian
  Version:                           1 (current)
  OS/ABI:                            UNIX - System V
  ABI Version:                       0
  Type:                              DYN (Shared object file)
  Machine:                           AArch64
  Version:                           0x1
  Entry point address:               0x2a500
  Start of program headers:          1888256 (bytes into file)
  Start of section headers:          954888 (bytes into file)
  Flags:                             0x0
  Size of this header:               64 (bytes)
  Size of program headers:           56 (bytes)
  Number of program headers:         9
  Size of section headers:           64 (bytes)
  Number of section headers:         24
  Section header string table index: 22   

However there is an issue - honestly I needed to report it before:

if you run patchelf --print-interpreter /tmp/astap

You get libdl.so and maybe it was correct someday nowadays it should be

patchelf --print-interpreter astap_cli

/system/bin/linker64 or /system/bin/linker

[artyom-beilis]

I used patchelf to change the path to correct one

[artyom-beilis]

Last thing if you are running it from application context and not stuff like termux here following tricks:

1. Put it into same directory like other jni libraries and rename it to libsomething.so like I do libastap_cli.so - also it is exe and not really so - it is in order to make sure it would be placed in location you run it
2. Add android:extractNativeLibs="true" to main manifest to make sure the exe is actually extracted. And add to app/build.gradle (I think you need both not sure)

   
   android {
   ...
   packagingOptions {
       jniLibs {
           useLegacyPackaging = true
       }
   }

}

3. To get path to the location of libs call `getApplicationInfo().nativeLibraryDir` 

Good luck

[laheller]

@artyom-beilis

The problem here I think is with the line:

Type: DYN (Shared object file) which is printed by readelf in case when the file is a shared library.

But the original intention for astap_cli (CLI for me suggest Command Line Interface) I think was to have it built as an executable, where readelf shoud print

Type: EXEC (Executable file)

Which it does for the older astap_cli versions for Android!

@han-k59 correct me if I am wrong. Anyway when I try the astap CLI built for Windows or Linux, they are really executables and I can call them from command line with parameters, etc.

The main problem is I think that the last versions of Android astap_cli files were built incorrectly as shared object instead of simple executables.

A difference is for example for C language GCC compiler is the below gcc foo.c -o foo => this creates an executable "foo" from the source file "foo.c" gcc -shared -o libfoo.so foo.o => this creates a shared library "libfoo.so" from the input object file "foo.o"

I don't know how this works for the FreePascal compiler but I think the problem should be somewhere at the build.

BR,

Ladislav

[artyom-beilis]

The problem here I think is with the line:

Type: DYN (Shared object file) which is printed by readelf in case when the file is a shared library

I use astap_cli as it is today with only "patching" of the interpreter

Note on Linux Ubuntu amd64

$ readelf -h /bin/cat 
ELF Header:
...
  ABI Version:                       0
  Type:                              DYN (Shared object file)
  Machine:                           Advanced Micro Devices X86-64
...

I think you mixing several things

Hello world pascal:
$fpc h.pas 
$ readelf -h ./h
Type:                              EXEC (Executable file)
$ file ./h
file ./h
./h: ELF 64-bit LSB executable,  ----> statically linked <-----

$ gcc h.c -o h_c
$readelf -h h_c 
Type:                              DYN (Shared object file)
$file h_c
 ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), ---> dynamically linked, <---

FreePascal by default create statically linked executable on Linux while gcc and most other compiles do dynamic linking - so the astap_cli for Android - create dynamically linked exe

[laheller]

@artyom-beilis

OK, but if you compare the last two released versions of astap_cli, the older one has Type = EXEC and the latest has Type = DYN. I guess both versions were compiled using fpc. The older one works for me without problems as an executable callable from terminal command line, while the latest one does not...

[artyom-beilis]

I guess both versions were compiled using fpc. The older one works for me without problems as an executable callable from terminal command line, while the latest one does not...

At least ones that I use for last 1/2 a year all are dynamically linked.

[laheller]

@artyom-beilis @han-k59

OK, so the latest Android version of astap_cli can be executed from terminal without any patching simply using the below command line:

linker64 /data/local/tmp/astap/astap_cli --help

Output is as expected:

ASTAP astrometric solver version CLI-2023.11.09 (C) 2018, 2022 by Han Kleijn. License MPL 2.0, Webpage: www.hnsky.org Usage: -f filename {fits, tiff, png, jpg files} -r radius_area_to_search[degrees] -z downsample_factor[0,1,2,3,4] {Downsample prior to solving. 0 is auto} -fov diameter_field[degrees] {enter zero for auto} -ra center_right_ascension[hours] -spd center_south_pole_distance[degrees] -s max_number_of_stars {default 500} -t tolerance {default 0.007} -m minimum_star_size["] {default 1.5} -check apply[y/n] {Apply check pattern filter prior to solving. Use for raw OSC images only when binning is 1x1} -speed mode[auto/slow] {Slow is forcing reading a larger area from the star database (more overlap) to improve detection} -o file {Name the output files with this base path & file name} -d path {specify a path to the star database} -D abbreviation {Specify a star database [d80,d50,..]} -analyse snr_min {Analyse only and report median HFD and number of stars used} -extract snr_min {As -analyse but additionally write a .csv file with the detected stars info} -log {Write the solver log to file} -progress {Log all progress steps and messages} -update {update the FITS header with the found solution. Jpeg, png, tiff will be written as fits} -wcs {Write a .wcs file in similar format as Astrometry.net. Else text style.} Preference will be given to the command line values.

Thanks @artyom-beilis for the hints! @han-k59, issue can be closed.

PS: Maybe a small feature request: The solver output could be also the console instead of disk files, like for example if the value of parameter -o is "-", then let's just print the plate solver results to the console.

[han-k59]

I see using the linker64 as a temperary solution. At the moment I can't fix the executable file problem. I assume I have to compile with the linker parameter no-pie. But it doesn't work.

The astap-cli versions after 2023-9-18 are faster. So it will be good to upgrade. At the moment I'm waiting for advice how to fix this.

Maybe a small feature request: The solver output could be also the console instead of disk files, like for example if the value of parameter -o is "-", then let's just print the plate solver results to the console.

I can add that but what about informative messages which also go to the console. Can you handle both the current messages plus the contains of the current .ini file to console??

Han

[artyom-beilis]

I see using the linker64 as a temperary solution.

You can run patchelf --set-interpreter /system/bin/linker64 astap_cli or /system/bin/linker for 32 bit system. This is what I did to make it run properly under Android.

Maybe you can consult FPC developers regarding default runtime linker?

[laheller]

@han-k59

I can add that but what about informative messages which also go to the console. Can you handle both the current messages plus the contains of the current .ini file to console??

Han

OK, good point. Maybe the informative stuff could go to the console standard output while the solver results to the console standard error stream.

BR,

Ladislav

[han-k59]

The problem is now fixed. The new binary versions are available at:

https://sourceforge.net/projects/astap-program/files/android/

This version should be faster then the once from June 2023.

Please tell me if they are working for you

Han

[laheller]

The problem is now fixed. The new binary versions are available at:

https://sourceforge.net/projects/astap-program/files/android/

This version should be faster then the once from June 2023.

Please tell me if they are working for you

Han

Hi @han-k59

Unfortunately it's even worse than before. When I try to run via the regular way by simple calling using filename: a54x:/ # /data/local/tmp/astap/astap_cli /system/bin/sh: /data/local/tmp/astap/astap_cli: No such file or directory

When I try via linker64: 126|a54x:/ # linker64 /data/local/tmp/astap/astap_cli error: "/data/local/tmp/astap/astap_cli" has unexpected e_type: 2

BR,

Ladislav

[han-k59]

Hi,

It rapports "No such file or directory". That indicates it could not find the file. Path wrong?

Did the version from June 2023 work for you?

Han

[laheller]

@han-k59

The file is there. The "no such file or directory" error message is misleading. Problem is probably still something similar mentioned by @artyom-beilis . Nevermind, I will try the method with patchelf and will let you know.

[laheller]

Hi @han-k59

When I apply the patchelf trick suggested by @artyom-beilis on the latest version of astap_cli (version 2024-06-10), I receive the following when I attempt to run it:

a54x:/data/local/tmp/astap # ./astap_cli "./astap_cli": error: Android 5.0 and later only support position-independent executables (-fPIE).

I know you are using a different compiler than gcc but the switch -fPIE is one used in gcc. Maybe there is an fpc equivalent...

[han-k59]

I will have a look.....

[han-k59]

According the forum the -k-pie compiler option is required. Strangly the executable is reported again as shared object by the command readelf -h ./astap_cli:

DYN (Shared object file)

Can you try if this works for you?

Download 64 bit version: https://sourceforge.net/projects/astap-program/files/android/astap_command-line_version_Android_aarch64.zip/download

[laheller]

According the forum the -k-pie compiler option is required. Strangly the executable is reported again as shared object by the command readelf -h ./astap_cli:

DYN (Shared object file)

Can you try if this works for you?

Download 64 bit version: https://sourceforge.net/projects/astap-program/files/android/astap_command-line_version_Android_aarch64.zip/download

@han-k59 It again needs either the patchelf or the _linker64 ./astapcli trick, but now it works....

Anyway, for me still the version 2022-04-23 is what works without any trick. Would be possible to rebuild the current version using the same way (command line switches, or the same toolchain) as the v2022-04-23 was built?

[han-k59]

I can't reproduce the way version 2022-04-23 was compiled. I will do some further investigation. This problem has to be solved.

If I remember the shared library versions run in the Bluestacks emulator. So this is difficult to understand. Do you test/run them in an Android phone/tablet or do you have an emulator?

[han-k59]

At the moment I don't have an Android system setup to test.

[laheller]

@han-k59

I have 3 different brand Android devices and have the same behaviour on each.

[han-k59]

I think the problem is in the cross-compiler. A simple "hello world" program I can compile to a Linux pie executable. However the same program cross compiles to Android as a shared object. This requires some further investigation and experimenting. I will try to find a fix.....

Previously I came to the conclusion that the 2022-04-23 version was compiled in an older cross compiler I do not have anymore.

[han-k59]

At the moment I waiting for so response on the forum. However it seems that a pie executable can be reported as a dynamic library. But both Lubuntu and Debian report astap_cli as a dynamic library.

https://forum.lazarus.freepascal.org/index.php/topic,56842.30.html https://forum.lazarus.freepascal.org/index.php/topic,39338.msg451236.html#msg451236

I tried in Lubuntu this : patchelf --set-interpreter /system/bin/linker64 ./astap_cli

I doesn't change anything. Has this to be done in Android?

[artyom-beilis]

Patch elf can be run on normal linux. That what I do.

You can check before and after calling

patchelf --print-interpreter /path/to/exe

For example for linux ls

$ patchelf --print-interpreter /bin/ls 
/lib64/ld-linux-x86-64.so.2

[han-k59]

Okay before: h@h-vmwarevirtualplatform:~/astap.fpc/command-line_version$ patchelf --print-interpreter ./astap_cli libdl.so

After ~/astap.fpc/command-line_version$ patchelf --print-interpreter ./astap_cli /system/bin/linker64

It is still a DYN (Shared object file) but if this is the fix to get it running in Android, then I only have to use this command to fix it.

So for 64 bit patchelf --set-interpreter /system/bin/linker64 ./astap_cli

For 32 bit patchelf --set-interpreter /system/bin/linker ./astap_cli

What patchelf command for the Android x86_64 version?

[artyom-beilis]

Same for x86 and x64 according to bits. It is because 64 bit systems allow running of both 32 and 64 bit.

[laheller]

@han-k59 @artyom-beilis

Problem solved. I took the sources and followed the guide to set up a development environment to build astap_cli.

Problem was the highlighted custom compiler option -Cg which means Generate PIC code:

I removed this option and rebuilt the astap_cli and voila, it works like a charm without any patching.

BR,

Ladislav

[han-k59]

Nice you have it compiling. But I see still see problems. The cpu option -CpARMV7A does not work. Secondly it produces a version as follows: patchelf --print-interpreter ./astap_cli libdl.so

So the interpreter is libdl.so. I understand from artyom-beilis that the interpreter should be /system/bin/linker64. So it should not work. Can you run the file ./astap_cli on the new file?

[han-k59]

At the moment I understand the options should as the screenshot below plus this patchelf command:

patchelf --set-interpreter /system/bin/linker64 ./astap_cli

[laheller]

Nice you have it compiling. But I see still see problems. The cpu option -CpARMV7A does not work. Secondly it produces a version as follows: patchelf --print-interpreter ./astap_cli libdl.so

So the interpreter is libdl.so. I understand from artyom-beilis that the interpreter should be /system/bin/linker64. So it should not work. Can you run the file ./astap_cli on the new file?

@han-k59 The cpu option -CpARMV7A was just my experiment, it's not required.

I built both aarch64 and arm versions of astap_cli and here is the readelf header output for the arm version:

a54x:/data/local/tmp/astap # readelf -h ./astap_cli ELF Header: Magic: 7f 45 4c 46 01 01 01 00 00 00 00 00 00 00 00 00 Class: ELF32 Data: 2's complement, little endian Version: 1 (current) OS/ABI: UNIX - System V ABI Version: 0 Type: DYN (Shared object file) Machine: arm

We can check the default interpreter directly using readelf the following way:

a54x:/data/local/tmp/astap # readelf -a ./astap_cli | grep interpreter [Requesting program interpreter: /system/bin/linker]

As you can see, in my case - without any patching - it points directly to the /system/bin/linker, which is correct. At the end I just removed the -Cg compiler option and it works. Also FYI, I executed both the above readelf commands directly on my Android device via the ADB (Android Debug Bridge) shell.

BR,

Ladislav

[laheller]

At the moment I understand the options should as the screenshot below plus this patchelf command:

patchelf --set-interpreter /system/bin/linker64 ./astap_cli

I don't think so. For me the -Cg and -k-pie are kind of conflicting and mutually exclusive options, where if both applied, apparently the -Cg seems to have a higher priority.

[han-k59]

Confusing on the Lazarus forum they always report -Cg -k-pie -k-znow. E.g. https://wiki.freepascal.org/hardening

Since you can test it on Android please confirm the correct settings. I can not test Android at the moment.

The following link reports that the interpreter has changed:

android.aarch64 uses an old libdl.so that was replaced by libdl.so.2 in modern distributions on aarch64 https://github.com/dashingsoft/pyarmor/issues/1674

So maybe the interpreter should be libdl.so.2 ??

[laheller]

@han-k59

Already tried all the combinations of the mentioned switched, like -Cg -k-pie -k-znow -k-pie -k-znow -Cg -k-znow -Cg -k-pie

If we assume that the result astap_cli binary should work out-of-the-box, without any patching, than the build WITHOUT -Cg was the correct choice on my Android device. All the other cases (using -Cg) I also must patch the result binary.

What I can do from now is to test your published build, no matter, how your create it. Anyway, I think we can close this issue.

BR,

Ladislav

[han-k59]

It has to works for any Android version. I understand for Android 5+ it has to be a pie executable which seems to be the same as a shared object. Anyhow after your first post, I added the option -Cg- to get an executable reported but this was likely a static executable which did not work for you.

If the produced file works without any option out of the box then lets try that. I will remove all options and recompile all again. Then tell me if this works. Hopefully so then we can close this. I will recompile now and report when it is finished.

[han-k59]

All Android versions are recompiled without any option. Tell me if this works out of the box:

https://sourceforge.net/projects/astap-program/files/android/

[han-k59]

The file is now reported as: ./astap_cli: ELF 64-bit LSB shared object, ARM aarch64, version 1 (SYSV), dynamically linked, interpreter libdl.so, BuildID[sha1]=8eeee3c498f7728fe5428122b4acb179362e7bc3, stripped

The interpreter is libdl.so. Does this also work for Artyom-Beilis or does the interpreter has to be modified to libdl.so.2 or /system/bin/linker64?

[han-k59]

As you can see, in my case - without any patching - it points directly to the /system/bin/linker, which is correct.

Not on my system. Okay I missed that. Why is this happening.....

[laheller]

All Android versions are recompiled without any option. Tell me if this works out of the box:

https://sourceforge.net/projects/astap-program/files/android/

@han-k59 I can only test the armhf and aarch64 versions and NONE OF THEM worked, as you wrote, both are expecting the libdl.so as interpreter which apparently is not accessible.

I just renamed the binaries to easily distinguish them:

126|a54x:/data/local/tmp/astap # readelf -a ./astap_cli_armhf | grep interpreter [Requesting program interpreter: libdl.so]

a54x:/data/local/tmp/astap # readelf -a ./astap_cli_aarch64 | grep interpreter [Requesting program interpreter: libdl.so]

Interesting that on your side that libdl.so is set as interpreter. Probably because of the different build environment.

[han-k59]

I compile them normally under Linux but also the LAMW version under Windows produces here files with as interpreter libdl.so.

Anyhow I have now added the patchelf --set-interpreter /system/bin/linker64 ./astap_cli command in my compile_all.sh file. So now they should be all good. Artyom-Beilis has already given the solution long time ago but I was on the wrong footing due to the confusion between pie executable and shared object.

So this topic can be closed if they now work out of the box. PLEASE CONFIRM!. https://sourceforge.net/projects/astap-program/files/

If this all work then I will close this issue and later find out how to define the interpreter without pathelf via the Lazarus forum. The problem is now enough clear :).

[artyom-beilis]

Interesting that on your side that libdl.so is set as interpreter. Probably because of the different build environment.

libld is standard on normal Linux, from what I understand Android requires different interpreter :-) anyway good you finally dig the right build config

[laheller]

@han-k59

I'll check later your rebuilt and patched binaries and will let you know.

[laheller]

So this topic can be closed if they now work out of the box. PLEASE CONFIRM!. https://sourceforge.net/projects/astap-program/files/

@han-k59 I can confirm astap_cli the armhf & aarch64, now both they work.

[han-k59]

Closed