search

dstndstn / astrometry.net

Astrometry.net -- automatic recognition of astronomical images
http://astrometry.net
Other
650 stars 184 forks source link

Netpbm location issues compiling 0.94 #276

Closed alexw-im closed 2 months ago

alexw-im commented 10 months ago

Related to #272 but not quite the same since I'm not using docker. I just recently realized that when I switched to 0.94 running on a raspberry pi, there seem to be path issues with netpbm during compilation. solve-field completes but the only output plot written is *-objs.jpg, and it's empty.

I manually created the symlink mentioned above on my pi before installing 0.94 and that seems to have fixed the issues below, that is ln -s /usr/include /usr/local/include/netpbm before make; make extra; sudo make install

Since there's no dockerfile for a bare metal install, I'm not sure where to fix this mine for future bare-metal users, but perhaps there's a good solution? A new check in the makefile or something?

For direct comparison, here's output from 0.93 with no errors and everything working per my expectations:

~/test-images $ solve-field --use-source-extractor --overwrite 019_21deg_gain5.jpg
Reading input file 1 of 1: "019_21deg_gain5.jpg"...
jpegtopnm: WRITING PPM FILE
Read file stdin: 1280 x 800 pixels x 1 color(s); maxval 255
Using 8-bit output
Extracting sources...

> WARNING: default.sex not found, using internal defaults

----- Source Extractor 2.25.0 started on 2023-08-24 at 14:12:07 with 1 thread

----- Measuring from: tmp.fits.eeKxqi
      "Unnamed" / no ext. header / 1280x800 / 8 bits (integers)
(M+D) Background: 51.2905    RMS: 4.09629    / Threshold: 6.14444
      Objects: detected 49       / sextracted 49

> All done (in 0.1 s: 5872.2 lines/s , 359.7 detections/s)
Sorting brightness using MAG_AUTO and BACKGROUND columns failed; falling back to MAG_AUTO.
Reading sort column "MAG_AUTO"
Sorting sort column
mmapping input file
Copying table header.
Writing row 0
Done
Solving...
Reading file "./019_21deg_gain5.axy"...
Field 1 did not solve (index index-4119.fits, field objects 1-10).
Field 1 did not solve (index index-4118.fits, field objects 1-10).
Field 1 did not solve (index index-4117.fits, field objects 1-10).
Field 1 did not solve (index index-4116.fits, field objects 1-10).
Field 1 did not solve (index index-4115.fits, field objects 1-10).
Field 1 did not solve (index index-4114.fits, field objects 1-10).
  log-odds ratio 137.689 (6.27208e+59), 18 match, 0 conflict, 9 distractors, 22 index.
  RA,Dec = (282.188,34.3405), pixel scale 14.336 arcsec/pix.
  Hit/miss:   Hit/miss: +++-+-+++++++-+-+++--++---+(best)-------------------+++
Field 1: solved with index index-4113.fits.
Field 1 solved: writing to file ./019_21deg_gain5.solved to indicate this.
Field: 019_21deg_gain5.jpg
Field center: (RA,Dec) = (282.186943, 34.345040) deg.
Field center: (RA H:M:S, Dec D:M:S) = (18:48:44.866, +34:20:42.145).
Field size: 5.09166 x 3.1871 degrees
Field rotation angle: up is -64.2145 degrees E of N
Field parity: neg
Creating new FITS file "./019_21deg_gain5.new"...
Creating index object overlay plot...
Creating annotation plot...
Your field contains:
  The star Sheliak (β Lyr / 10 Lyr)
  The star ν Lyr / ν2 Lyr / 9 Lyr
  The star ν1 Lyr / 8 Lyr

Here's the output with exactly the same environment, but using the 0.94 release package from github:

~/test-images $ solve-field --use-source-extractor --overwrite 019_21deg_gain5.jpg
Reading input file 1 of 1: "019_21deg_gain5.jpg"...
jpegtopnm: WRITING PPM FILE
Read file stdin: 1280 x 800 pixels x 1 color(s); maxval 255
Using 8-bit output
Extracting sources...

> WARNING: default.sex not found, using internal defaults

----- Source Extractor 2.25.0 started on 2023-08-24 at 14:29:56 with 1 thread

----- Measuring from: tmp.fits.XSj44y
      "Unnamed" / no ext. header / 1280x800 / 8 bits (integers)
(M+D) Background: 51.2905    RMS: 4.09629    / Threshold: 6.14444
      Objects: detected 49       / sextracted 49

> All done (in 0.1 s: 5713.3 lines/s , 349.9 detections/s)
Sorting brightness using MAG_AUTO and BACKGROUND columns failed; falling back to MAG_AUTO.
Reading sort column "MAG_AUTO"
Sorting sort column
mmapping input file
Copying table header.
Writing row 0
Done
cairoutils.c:715:cairoutils_read_ppm_stream: Netpbm is not available; can't read PPM imagescairoutils.c:715:cairoutils_read_ppm_stream: Netpbm is not available; can't read PPM images
cairoutils.c:715:cairoutils_read_ppm_stream: Netpbm is not available; can't read PPM images

cairoutils.c:715:cairoutils_read_ppm_stream: Netpbm is not available; can't read PPM imageslibpng warning: Image width is zero in IHDR
libpng warning: Image height is zero in IHDR
libpng error: Invalid IHDR data

Aborted
Solving...
Reading file "./019_21deg_gain5.axy"...
Field 1 did not solve (index index-4119.fits, field objects 1-10).
Field 1 did not solve (index index-4118.fits, field objects 1-10).
Field 1 did not solve (index index-4117.fits, field objects 1-10).
Field 1 did not solve (index index-4116.fits, field objects 1-10).
Field 1 did not solve (index index-4115.fits, field objects 1-10).
Field 1 did not solve (index index-4114.fits, field objects 1-10).
  log-odds ratio 137.689 (6.27208e+59), 18 match, 0 conflict, 9 distractors, 22 index.
  RA,Dec = (282.188,34.3405), pixel scale 14.336 arcsec/pix.
  Hit/miss:   Hit/miss: +++-+-+++++++-+-+++--++---+(best)-------------------+++
Field 1: solved with index index-4113.fits.
Field 1 solved: writing to file ./019_21deg_gain5.solved to indicate this.
Field: 019_21deg_gain5.jpg
Field center: (RA,Dec) = (282.186943, 34.345040) deg.
Field center: (RA H:M:S, Dec D:M:S) = (18:48:44.866, +34:20:42.145).
Field size: 5.09166 x 3.1871 degrees
Field rotation angle: up is -64.2145 degrees E of N
Field parity: neg
Creating new FITS file "./019_21deg_gain5.new"...
solve-field.c:327:plot_source_overlay Plotting command was cancelled
 solve-field.c:133:run_command Command was: "/usr/local/astrometry/bin/plotxy -I /tmp/tmp.ppm.0j8rPy -i ./019_21deg_gain5.axy -X X_IMAGE -Y Y_IMAGE -C red -w 2 -N 50 -x 1 -y 1 -P | /usr/local/astrometry/bin/plotxy -i ./019_21deg_gain5.axy -X X_IMAGE -Y Y_IMAGE -I - -w 2 -r 3 -C red -n 50 -N 200 -x 1 -y 1 > ./019_21deg_gain5-objs.png"

 solve-field.c:132:run_command Command exited with exit status 134

Here's the script I'm using to install astrometry generally:

# Install Astrometry.net
echo "Installing Astrometry.net 0.94"
wget -nc https://github.com/dstndstn/astrometry.net/releases/download/0.94/astrometry.net-0.94.tar.gz
tar -xzvf astrometry.net-0.94.tar.gz
cd astrometry.net-0.94

sudo apt update
sudo apt install -y build-essential curl git file pkg-config swig libcairo2-dev libnetpbm10-dev netpbm libpng-dev libjpeg-dev zlib1g-dev libbz2-dev libcfitsio-dev wcslib-dev python3 python3-pip python3-distutils python3-dev python3-scipy python3-pil python3-venv libatlas-base-dev

pip3 install numpy pyfits

make
make extra
sudo make install
echo "export PATH=/usr/local/astrometry/bin:$PATH" >> ~/.profile
sudo echo "export PATH=/usr/local/astrometry/bin:$PATH" >> /root/.profile
cd ../
# Back in install dir
./catalog-install.sh

Going from the (failed) 0.94 installation to the (good) 0.93 installation, all I did was sudo mv /usr/local/astrometry /usr/local/astrometry-0.94 and then remove astrometry.net-0.94.tar.gz, download the 0.93 release from http://astrometry.net/downloads/, and untar/make/make extra/sudo make install (and re-copy index files of course).

dstndstn commented 10 months ago

Hi,

This is a pain because netpbm is not packaged the same way on different systems/distributions.

If pkg-config exists, the Astrometry.net Makefile will use it to find netpbm. (In util/makefile.netpbm:)

NETPBM_INC ?= $(shell pkg-config --exists netpbm && pkg-config --cflags netpbm || echo)
NETPBM_LIB ?= $(shell pkg-config --exists netpbm && pkg-config --libs netpbm || echo "-L. -lnetpbm")

If pkg-config does not know about netpbm on your computer, you can instead set the NETPBM_INC and NETPBM_LIB environment variables, eg

export NETPBM_INC="-I/usr/include"
export NETPBM_LIB="-lnetpbm"

The other thing that can be different is, the astrometry.net code looks for the file netpbm/pam.h relative to the included directory -- so if the file /usr/include/netpbm/pam.h exists, it should work fine, while if you only have the file /usr/include/pam.h, then that lack of the netpbm directory causes a problem (and that's what the symlink trick can fix).

alexw-im commented 10 months ago

Very enlightening, thank you!

So all my systems do have pkg-config (it's in that install script). Interestingly, I notice that on an older installation with a fully-working copy of astrometry 0.93, pkg-config --exists netpbm returns nothing and 

$ pkg-config --cflags netpbm
Package netpbm was not found in the pkg-config search path.
Perhaps you should add the directory containing `netpbm.pc'
to the PKG_CONFIG_PATH environment variable
No package 'netpbm' found

I noticed you mention those environment variables in a different context but I don't quite understand how they're supposed to work, so it's possible I wasn't attempting that fix the right way: I kind of blindly jammed them into my .bashrc, but compiling 0.94, I still had the same problem above.

I guess the question is, what's different between 0.93 and 0.94 that make 0.93 finds netpbm on my system (despite pkg-config failing to locate it AND not having the symlink fix in place) where 0.94 fails?

dstndstn commented 10 months ago

This change https://github.com/dstndstn/astrometry.net/commit/d4760e2d25aa90b2817a02610b8daf10235b3cfb

On Thu, Aug 24, 2023 at 7:14 PM alexw-im @.***> wrote:

Very enlightening, thank you!

So all my systems do have pkg-config (it's in that install script). Interestingly, I notice that on an older installation with a fully-working copy of astrometry 0.93, pkg-config --exists netpbm returns nothing and

$ pkg-config --cflags netpbm Package netpbm was not found in the pkg-config search path. Perhaps you should add the directory containing `netpbm.pc' to the PKG_CONFIG_PATH environment variable No package 'netpbm' found

I noticed you mention those environment variables in a different context but I don't quite understand how they're supposed to work, so it's possible I wasn't attempting that fix the right way: I kind of blindly jammed them into my .bashrc, but compiling 0.94, I still had the same problem above.

I guess the question is, what's different between 0.93 and 0.94 that make 0.93 finds netpbm on my system (despite pkg-config failing to locate it AND not having the symlink fix in place) where 0.94 fails?

— Reply to this email directly, view it on GitHub https://github.com/dstndstn/astrometry.net/issues/276#issuecomment-1692530489, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAIEH7OCJFI4RJISNPYAPATXW7N35ANCNFSM6AAAAAA35VETQA . You are receiving this because you commented.Message ID: @.***>

alexw-im commented 10 months ago

Ahh well that seems pretty clear! So I guess the idea is that this breaks install on i.e. a raspberry pi but improves likelihood of success on other systems?

dstndstn commented 10 months ago

Yes