Missing some inbuilt .pro files and examples folder

[centarsirius]

Platform - Ubuntu budgie 18.04 LTS

My gnudatalanguage folder does not consist of the "examples folder", all the .pro files are stored at /usr/share/gnudatalanguage/lib , but these are inbuilt procedure files such as factorial.pro, appleman.pro. I have been searching all of my system for those examples but all in vain. Some inbuilt .pro files too are missing such as binomial.pro, cramer.pro

i used sudo apt-get install gnudatalanguage to install GDL as given here

[alaingdl]

Hi. Few points in your post are not clear for me.

• directory /usr/share/doc/gnudatalanguage is not really useful (first snapshot)
• directory /usr/share/gnudatalanguage/lib is much more useful since it should include all the functions and procedures we (collectively) wrote in the IDL syntax. We don't have all func/pro provided by IDL but we can add contributions as long as the code is under a free licence like GPL or BSD. See #737 which provided 2 codes now in the Git
• To list the built-in functions/pro, you can do in GDL : HELP,/lib
• It seems that you mix together some *.pro files from GDL with some other files from Coyote. I would have prefer to keep the Coyote's one in another folder
• If you have the *pro files provided by IDL around (in a givent path) you can add that path into your GDL_PATH (it should appears later with !path)

I strongly advice you to update (compile by yourself) the current Git version (1.0.0rc2) since 0.9.7 packaged in (bionic) u18.04 is quite old https://launchpad.net/ubuntu/bionic/+package/gnudatalanguage A summary of instructions is in #749

Concerning BINOMIAL() and CRAMER() they are not in GDL now. But for BINOMIAL, since we do have LNGAMMA & GAUSS_PDF (this /gauss keyword disappeared in IDL 8.7 ?!) I think it is not a huge work to add it. Rewrite a CRAMER() by our-self should be super fast, itn't it just related to matrix inversion ? We do have INVERT() & DETERM().

[centarsirius]

i did follow #749 and thanks to @mvandam 's quick summary i was able to upgrade to version 1.0.0rc2 and i now understand that there are issues with directly copying IDL's code so it takes time for you guys to develop .pro files

It seems that you mix together some *.pro files from GDL with some other files from Coyote. I would have prefer to keep the Coyote's one in another folder

can that be undone collectively or do i have to find each file and move them?

and new errors have emerged: (i don't use linux that much so this may look dumb)

centarsirius@sachin-Legion-Y540:~$ gdl
> gdl: /home/centarsirius/anaconda3/lib/libtiff.so.5: no version information available (required by gdl)
> gdl: /home/centarsirius/anaconda3/lib/libtiff.so.5: no version information available (required by /usr/lib/x86_64-linux-gnu/libwx_gtk3u_core-3.0.so.0)
> gdl: /home/centarsirius/anaconda3/lib/libtiff.so.5: no version information available (required by /usr/lib/x86_64-linux-gnu/libgeotiff.so.5)
> gdl: /home/centarsirius/anaconda3/lib/libtiff.so.5: no version information available (required by /usr/lib/libGraphicsMagick-Q16.so.3)
  GDL - GNU Data Language, Version 1.0.0-rc.2 git

what are these?

[mvandam]

I had similar issues to you on a machine where I had installed anaconda as user (not root). What I did to get around the issues with the anaconda installation is to build gdl as root (so it did not know anything about anaconda):

sudo cmake .. -DGRIB=off -DPYTHON=off sudo make -j 4 sudo make install

This worked for me (I will add this information to #749)

[alaingdl]

• Anaconda ? I would say you have an anaconda env active around and it is dangerous ! (please use a clean terminal and a clean shell without anaconda prefixed in the PATH !)

• did you run "make check" at the end of compilation, in the build/ directory, to be sure most is OK ? (only very few tests may failed)

• if you have a good 1.0.0rc2, you can remove the old files, using the files in src/pro (copy somewhere or just set it in GDL_PATH) and you can download again the last Coyote & co (in the Debian git)

[alaingdl]

I ate that Anaconda did change my .bashrc (then create mess around) then I source by myself when I need to activate conda ... in a new/another terminal (the block starting by :

 > # >>> conda initialize >>>

[centarsirius]

@alaingdl i have anaconda in path because it takes care of almost all my dependencies and package needs as i mostly use python. Had it this way for about a year now, no problems except sometimes nothing can be opened from anaconda-navigator (different problem with some error in syntax). no, i didn't run make check, i didnt even know it existed (sorry!) ran it now and 5/232 tests failed, that would be ok, right?

98% tests passed, 5 tests failed out of 232

Total Test time (real) =  87.29 sec

The following tests FAILED:
     98 - test_fft_leak.pro (Failed)
    107 - test_file_test.pro (Failed)
    110 - test_fix.pro (Failed)
    122 - test_hdf5.pro (Failed)
    142 - test_make_dll.pro (Failed)
Errors while running CTest
make[4]: *** [Makefile:118: test] Error 8
make[3]: *** [testsuite/CMakeFiles/check.dir/build.make:59: testsuite/CMakeFiles/check] Error 2
make[2]: *** [CMakeFiles/Makefile2:222: testsuite/CMakeFiles/check.dir/all] Error 2
make[1]: *** [CMakeFiles/Makefile2:229: testsuite/CMakeFiles/check.dir/rule] Error 2
make: *** [Makefile:214: check] Error 2

and there's no pro folder in src directory

@mvandam does that pose any risk or problems while doing work in GDL? do i have to rebuild or i can leave it as it is?

[mvandam]

I failed these three: 98 - test_fft_leak.pro (Failed) 117 - test_get_screen_size.pro (Failed) 142 - test_make_dll.pro (Failed) I have not looked into this.

[alaingdl]

• tests : only 5 ? perfect ! You can run manually using $ sh ../quick_start_GDL.sh to look at the details (verbose ...)

• pro files are in sub dir. src/pro/ (or make $ find ~ -iname '*.pro')

[centarsirius]

so if i rebuild it with

#as root
sudo cmake .. -DGRIB=off -DPYTHON=off
sudo make -j 4
sudo make install

i won't be making a duplicate right? this build would replace the existing build?

[mvandam]

Yes, it would replace the existing one.

[centarsirius]

centarsirius@sachin-Legion-Y540:~$ sudo -s
root@sachin-Legion-Y540:/home/centarsirius# cd gdl
root@sachin-Legion-Y540:/home/centarsirius/gdl# cd build
root@sachin-Legion-Y540:/home/centarsirius/gdl/build# sudo cmake .. -DGRIB=off -DPYTHON=off
CMake Deprecation Warning at CMakeLists.txt:22 (cmake_policy):
  The OLD behavior for policy CMP0026 will be removed from a future version
  of CMake.

  The cmake-policies(7) manual explains that the OLD behaviors of all
  policies are deprecated and that a policy should be set to OLD only under
  specific short-term circumstances.  Projects should be ported to the NEW
  behavior and not rely on setting a policy to OLD.

-- INFO: will use GNU extensions for STDIO (useful for compressed I/O) since it seems accepted by your c++ compiler.
-- INFO: We prefer to use Readline than Editline.
Feedback welcome on OSX Readline ...
Use fuzzy detection for PLplot lib. (e.g. in /usr/lib)
-- Warning, if you have plplot version > 5.11 and wxWidgets enabled, please read carefully file README
-- Using a plplot library without private functions - workarounds will be used.
-- INFO: We prefer to use GraphicsMagick than ImageMagick
-- WARNING: Dynamic drivers may have to be disabled in plplot.
To make plplot work with ImageMagick:
use -DENABLE_DYNDRIVERS=OFF cmake option when compiling plplot or
use -DMAGICK=OFF here to not use ImageMagick in GDL.
-- HDF5: Using hdf5 compiler wrapper to determine C configuration
-- Summary

GDL - GNU DATA LANGUAGE [Standalone]
System                 Linux-5.4.0-29-generic
Files generated        Unix Makefiles
GDL output             /home/centarsirius/gdl/build/src/gdl
Installation prefix    /usr/local
C++ compiler           /usr/bin/c++ -O3 -DNDEBUG

-- Options
OpenMP support         ON (flag: -fopenmp)
WxWidgets              ON (libs:-L/usr/lib/x86_64-linux-gnu;-pthread;-lwx_baseu-3.0;-lwx_gtk3u_core-3.0;-lwx_gtk3u_adv-3.0; headers:)
ImageMagick            OFF
TIFF                   ON (libs:/usr/lib/x86_64-linux-gnu/libtiff.so; headers:/usr/include/x86_64-linux-gnu)
GeoTIFF                ON (libs:/usr/lib/x86_64-linux-gnu/libgeotiff.so; headers:/usr/include/geotiff)
NetCDF                 ON (libs:netcdf; headers:)
HDF4                   ON (libs:/usr/lib/libmfhdfalt.so;/usr/lib/libdfalt.so;z;/usr/lib/x86_64-linux-gnu/libjpeg.so; headers:/usr/include/hdf)
HDF5                   ON (libs:/home/centarsirius/anaconda3/lib/libhdf5.so;/usr/lib/x86_64-linux-gnu/librt.so;/usr/lib/x86_64-linux-gnu/libpthread.so;/home/centarsirius/anaconda3/lib/libz.so;/usr/lib/x86_64-linux-gnu/libdl.so;/usr/lib/x86_64-linux-gnu/libm.so; headers:/home/centarsirius/anaconda3/include)
FFTW                   ON (libs:/usr/lib/x86_64-linux-gnu/libfftw3.so;/usr/lib/x86_64-linux-gnu/libfftw3f.so; headers:/usr/include)
Libproj4               ON (libs:/usr/lib/x86_64-linux-gnu/libproj.so; headers:/usr/include)
MPI                    OFF
Python                 OFF
UDUNITS-2              ON (libs:/usr/lib/x86_64-linux-gnu/libudunits2.so; headers:/usr/include)
EIGEN3                 ON (libs:; headers:/usr/include/eigen3)
GRAPHICSMAGICK         ON (libs:/usr/lib/libGraphicsMagick.so;/usr/lib/libGraphicsMagick++.so; headers:/usr/include/GraphicsMagick)
GRIB                   OFF
GLPK                   ON (libs:/usr/lib/x86_64-linux-gnu/libglpk.so; headers:/usr/include)
SHAPELIB               ON (libs:/usr/lib/x86_64-linux-gnu/libshp.so; headers:/usr/include)
EXPAT                  ON (libs:/usr/lib/x86_64-linux-gnu/libexpat.so; headers:/usr/include)
pslib                  ON (libs:/usr/lib/x86_64-linux-gnu/libps.so; headers:/usr/include)
Xlib                   ON (libs:/usr/lib/x86_64-linux-gnu/libSM.so;/usr/lib/x86_64-linux-gnu/libICE.so;/usr/lib/x86_64-linux-gnu/libX11.so;/usr/lib/x86_64-linux-gnu/libXext.so; headers:/usr/include)
libpng                 ON (libs:; headers:)

-- Mandatory modules
Plplot                 ON (libs:/usr/lib/x86_64-linux-gnu/libplplot.so;/usr/lib/x86_64-linux-gnu/libplplotcxx.so; headers:/usr/include)
GNU Readline           ON (libs:/usr/lib/x86_64-linux-gnu/libreadline.so;/usr/lib/x86_64-linux-gnu/libhistory.so; headers:/usr/include)
BSD Editline           OFF
GSL                    ON (libs:/usr/lib/x86_64-linux-gnu/libgsl.so;/usr/lib/x86_64-linux-gnu/libgslcblas.so; headers:/usr/include)
Zlib                   ON (libs:/usr/lib/x86_64-linux-gnu/libz.so; headers:/usr/include)
RPC                    ON (libs:; headers:/usr/include)
(N)curses              ON (libs:/usr/lib/x86_64-linux-gnu/libncurses.so;/usr/lib/x86_64-linux-gnu/libform.so; headers:/usr/include)

-- Configuring done
CMake Warning at src/CMakeLists.txt:203 (add_executable):
  Cannot generate a safe runtime search path for target gdl because files in
  some directories may conflict with libraries in implicit directories:

    runtime library [libncurses.so] in /usr/lib/x86_64-linux-gnu may be hidden by files in:
      /home/centarsirius/anaconda3/lib
    runtime library [libform.so.6] in /usr/lib/x86_64-linux-gnu may be hidden by files in:
      /home/centarsirius/anaconda3/lib
    runtime library [libreadline.so.8] in /usr/lib/x86_64-linux-gnu may be hidden by files in:
      /home/centarsirius/anaconda3/lib
    runtime library [libhistory.so.8] in /usr/lib/x86_64-linux-gnu may be hidden by files in:
      /home/centarsirius/anaconda3/lib
    runtime library [libz.so.1] in /usr/lib/x86_64-linux-gnu may be hidden by files in:
      /home/centarsirius/anaconda3/lib
    runtime library [libpng16.so.16] in /usr/lib/x86_64-linux-gnu may be hidden by files in:
      /home/centarsirius/anaconda3/lib
    runtime library [libgslcblas.so.0] in /usr/lib/x86_64-linux-gnu may be hidden by files in:
      /home/centarsirius/anaconda3/lib
    runtime library [libtiff.so.5] in /usr/lib/x86_64-linux-gnu may be hidden by files in:
      /home/centarsirius/anaconda3/lib
    runtime library [libfftw3.so.3] in /usr/lib/x86_64-linux-gnu may be hidden by files in:
      /home/centarsirius/anaconda3/lib
    runtime library [libfftw3f.so.3] in /usr/lib/x86_64-linux-gnu may be hidden by files in:
      /home/centarsirius/anaconda3/lib
    runtime library [libexpat.so.1] in /usr/lib/x86_64-linux-gnu may be hidden by files in:
      /home/centarsirius/anaconda3/lib
    runtime library [libSM.so.6] in /usr/lib/x86_64-linux-gnu may be hidden by files in:
      /home/centarsirius/anaconda3/lib
    runtime library [libICE.so.6] in /usr/lib/x86_64-linux-gnu may be hidden by files in:
      /home/centarsirius/anaconda3/lib
    runtime library [libX11.so.6] in /usr/lib/x86_64-linux-gnu may be hidden by files in:
      /home/centarsirius/anaconda3/lib
    runtime library [libXext.so.6] in /usr/lib/x86_64-linux-gnu may be hidden by files in:
      /home/centarsirius/anaconda3/lib

  Some of these libraries may not be found correctly.

-- Generating done
-- Build files have been written to: /home/centarsirius/gdl/build

i think i'm running into some duplication or version control problems here and should i use imagemagick or graphicmagick as suggested in the code? I need python support so i built with DPYTHON=ON this time and it gave me

-- Found PythonLibs: /usr/lib/x86_64-linux-gnu/libpython3.8.so (found suitable version "3.8.2", minimum required is "3") 
Traceback (most recent call last):
  File "<string>", line 1, in <module>
ModuleNotFoundError: No module named 'numpy'
CMake Error at CMakeLists.txt:892 (message):
  Python numpy package was not found.

  Use -DPYTHON=OFF to disable Python support.

how to overcome that?

[alaingdl]

I am not very "happy" to see you compiled GDL as Sudo. Except to add "official" packages, we made our best to avoid to be root to install it. And I installed it on HPCs where I was not root, and it is used by a large set of end-users.

[alaingdl]

• Concerning the message at the end of cmake, I think it is due to remaining conflicts with Anaconda. Please check whether anaconda is still in your PATH

• We has some troubles with ImageMackick in the past, long time ago. My experience was GraphicMagick behavior is more "predictable".

• for Python and Numpy, maybe @slayoo may have an idea ? (he recently made a change)

• I would say : lest goe ahead without python, then come back and activate it

[centarsirius]

i too am not happy with the root build, i know that using root unnecessarily can cause trouble, but i needed a problem free GDL, so i risked it and still it failed the same tests and the same error messages pop up on starting GDL

and yes anaconda is still in path

# <<< conda initialize <<<
export PATH=/home/centarsirius/anaconda3/bin:$PATH

[alaingdl]

• the few tests which are not OK are totally a detail, not important at all.
• which "error messages pop up on starting GDL" ?? I did not see it

[centarsirius]

these

centarsirius@sachin-Legion-Y540:~$ gdl
gdl: /home/centarsirius/anaconda3/lib/libtiff.so.5: no version information available (required by gdl)
gdl: /home/centarsirius/anaconda3/lib/libtiff.so.5: no version information available (required by /usr/lib/x86_64-linux-gnu/libwx_gtk3u_core-3.0.so.0)
gdl: /home/centarsirius/anaconda3/lib/libtiff.so.5: no version information available (required by /usr/lib/x86_64-linux-gnu/libgeotiff.so.5)
gdl: /home/centarsirius/anaconda3/lib/libtiff.so.5: no version information available (required by /usr/lib/libGraphicsMagick-Q16.so.3)
  GDL - GNU Data Language, Version 1.0.0-rc.2 git

edit: i went further and got this

centarsirius@sachin-Legion-Y540:~$ sudo find / -name "libtiff.so"
find: ‘/proc/37060/task/37060/net’: Invalid argument
find: ‘/proc/37060/net’: Invalid argument
/usr/lib/x86_64-linux-gnu/libtiff.so
/usr/local/comsol51/multiphysics/ext/graphicsmagick/glnxa64/libtiff.so
/home/centarsirius/anaconda3/pkgs/libtiff-4.1.0-hc7e4089_6/lib/libtiff.so
/home/centarsirius/anaconda3/pkgs/libtiff-4.1.0-h2733197_0/lib/libtiff.so
/home/centarsirius/anaconda3/lib/libtiff.so
find: ‘/run/user/1000/doc’: Permission denied
find: ‘/run/user/1000/gvfs’: Permission denied

[alaingdl]

• what is the content of your PATH ?

  $ echo $PATH

  if it did not contain "/home/centarsirius/anaconda3/" or "$HOME/anaconda3/" could you give $LD_LIBRARY_PATH. For my understanding of the pb, we cannot go ahead if they contain "anaconda3".

[centarsirius]

centarsirius@sachin-Legion-Y540:~$ echo $PATH
/home/centarsirius/Downloads/Astrosat_CZTI/czti_pipeline/czti/bin:
/home/centarsirius/anaconda3/bin:
/home/centarsirius/anaconda3/condabin:
/usr/local/sbin:
/usr/local/bin:
/usr/sbin:
/usr/bin:
/sbin:
/bin:
/usr/games:
/usr/local/games:
/snap/bin
centarsirius@sachin-Legion-Y540:~$ $LD_LIBRARY_PATH
bash: /home/centarsirius/Downloads/Astrosat_CZTI/czti_pipeline/czti/lib: Is a directory

[alaingdl]

As expected. You should remove /home/centarsirius/anaconda3/bin: & /home/centarsirius/anaconda3/condabin: from you path to go ahead in GDL. If you stay in the same terminal, just a copy should be enough, like (assuming Bash)

export PATH=/home/centarsirius/Downloads/Astrosat_CZTI/czti_pipeline/czti/bin:/usr/local/sbin: /usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin

ethen checking again if $PATH is clean...

then doing again the compilation in this clean env.

[centarsirius]

i did that, commented out the anaconda codes in .bashrc file, ran

export PATH=/home/centarsirius/Downloads/Astrosat_CZTI/czti_pipeline/czti/bin:/usr/local/sbin:
/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin

then went back to compiling, everything done and it still shows the same thing, i just learned that it's not an error but a warning. so i reset everything back. I'm going to use GDL this way until next patch or update comes out.

The problem for these warnings seems to be version mismatch as suggested here and here Please keep this issue open for 2 days, if there's something wrong, i will report it or else i will close it myself, got some work today in python, so won't be using GDL today. Thank you for all the help @alaingdl and @mvandam

[slayoo]

The CMake output seem to say that NumPy is not installed for the Python detected by CMake... my guess is that there are multiple Pythons available and it picks the one without NumPy?

Note that there are two distinct aspects of "Python support" in GDL - calling Python from GDL (which is OFF in the above CMake summary), and calling GDL from Python which requires compilind GDL as a Python module (i.e., -DPYTHON_MODULE=ON).

[GillesDuvert]

Hi, Just a comment on the !PATH. IDL/GDL users usually follow the recommendations such as found in idlastro and install their 3rd-party procedures in a (private, as IDL can be installed by the system) separate directory tree. The root of this tree (say, "/home/memyselfandI/idlastro") is to be added to the!PATHafter the IDL (or GDL) path by !PATH = !PATH+':'+expand_path('+/home/memyselfandI/idlastro') the expand_path function taking care of adding each subdirectories. Note the extra "+" before the /home. This makes maintenance easy. GDL follows IDL requirements that it starts by reading the commands in a file pointed to by the environment variable IDL_STARTUP (but also GDL_STARTUP). So the above PATH definition can be put in such a $IDL_STARTUP file and all the 3rd party procedures will be available both in IDL and GDL. People that have access to IDL's genuine procedures that have no counterpart in GDL can in principle add them in the GDL !PATH, preferably in last !PATH position. Many will "work" with GDL, admitedly not as fast.

[centarsirius]

@slayoo i thought it was the other way around, i want to call GDL from python sometimes, but both support would be an added benefit. and i checked for multiple versions of python but no only one is installed under conda

centarsirius@sachin-Legion-Y540:~$ which python
/home/centarsirius/anaconda3/bin/python
centarsirius@sachin-Legion-Y540:~$ cd /
centarsirius@sachin-Legion-Y540:/$ which python
/home/centarsirius/anaconda3/bin/python
centarsirius@sachin-Legion-Y540:/$ pip install numpy
Requirement already satisfied: numpy in /home/centarsirius/anaconda3/lib/python3.7/site-packages (1.18.4)

and i still am getting the error message that numpy is not detected

[slayoo]

When searching for NumPy, we are simply invoking

python -c "import numpy; print (numpy.get_include())"

(https://github.com/gnudatalanguage/gdl/blob/master/CMakeModules/FindNumpy.cmake#L25)

Could you please check what it returns on your system (from the same env as you run CMake). Thanks

[centarsirius]

this is what i got

centarsirius@sachin-Legion-Y540:~$ python -c "import numpy; print (numpy.get_include())"
/home/centarsirius/anaconda3/lib/python3.7/site-packages/numpy/core/include

same as above

[slayoo]

Very puzzling! Could you please try to print from CMake what happens around line 25 of FindNumpy.cmake - it seems that the numpy check behaves differently from within CMake

[centarsirius]

IF(PYTHON_EXECUTABLE)
    execute_process (COMMAND "${PYTHON_EXECUTABLE}" "-c" "import numpy; print (numpy.get_include())"
      OUTPUT_VARIABLE PYTHON_NUMPY_INCLUDE_DIR
      RESULT_VARIABLE PYTHON_NUMPY_NOT_FOUND)

I don't understand why this isn't detecting, that's the same thing we do to check numpy version. so i ran the same in python to check once again

#python 3.7.6
>>> import numpy
>>> print(numpy.get_include())
/home/centarsirius/anaconda3/lib/python3.7/site-packages/numpy/core/include

[slayoo]

Please try to print from CMake the PYTHON_NUMPY_INCLUDE_DIR variable (using the message() function) - perhaps this will yield a hint?

[centarsirius]

Sorry @slayoo this is the first time i'm using CMake but i still tried to add a new string entry as numpy_test with value message( STATUS "numpy is "${ python_numpy_include_dir}) but it didn't give me anything, attaching these in hopes that they help CMakeError.log CMakeOutput.log

[slayoo]

Perhaps please try recreating the makefiles (or even removing the whole build tree) after adding the messsage() command?

[alaingdl]

1/ GDL, Anaconda & Numpy

• current git clone of GDL; debian 10, without Anaconda env. grep -ni PYTHON_NUMPY_INCLUDE_DIR CMakeCache.txt 635:PYTHON_NUMPY_INCLUDE_DIR:STRING=/usr/lib/python3/dist-packages/numpy/core/include

compilation OK, test_python OK

• current git clone of GDL; debian 10, with an Anaconda env. grep -ni PYTHON_NUMPY_INCLUDE_DIR CMakeCache.txt compilation failed at link Linking CXX executable gdl /usr/bin/ld: /usr/lib/x86_64-linux-gnu/libSM.so: undefined reference to uuid_generate@UUID_1.0' /usr/bin/ld: /usr/lib/x86_64-linux-gnu/libSM.so: undefined reference touuid_unparse_lower@UUID_1.0' collect2: error: ld returned 1 exit status

To my opinion, one time again, to avoid problem(s), avoid compiling GDL within python provided by Anaconda

2/ reading your logs., maybe you have a pb related to Grib too. Do you have Grib around ? Do you use $ cmake .. -DGRIB=off ??

[centarsirius]

OK guys, it seems like i figured out the problem, compiling requires minimum python 3.8 (3.8.1 and above work better) and numpy version can be any recent versions, but the latest 1.18.4 works better. python 3.7.6 is installed in my conda base env so i installed python 3.8.2 outside it and installed numpy. you have to have installed complete numpy to compile python support modules.

Please note and check the following:

  * The Python version is: Python3.8 from "/usr/bin/python3"
  * The NumPy version is: "1.18.4"

and make sure that they are the versions you expect.
Please carefully study the documentation linked above for further help.

Original error was: No module named 'numpy.core._multiarray_umath'

since anaconda is my work env and it doesnt have complete support for 3.8 yet along with all the latest ppackages, i'm gonna stick to it until the update is released. As for the libtiff error, it's simply because of version mismatch.

[slayoo]

Python 3.8 is certainly not required. Here's yesterday's build log of master branch with Python 3.5: https://travis-ci.org/github/gnudatalanguage/gdl/jobs/686488215

[centarsirius]

i started this thread with 1 issue and on the way encountered two more and as @alaingdl already suggested to make on the base python, i should have done so, because i did that now and i encountered no errors except make check is not working

centarsirius@sachin-Legion-Y540:~/gdl/build$ cmake .. -DGRIB=off -DPYTHON=on -DPYTHON_MODULE=on
CMake Deprecation Warning at CMakeLists.txt:22 (cmake_policy):
  The OLD behavior for policy CMP0026 will be removed from a future version
  of CMake.

  The cmake-policies(7) manual explains that the OLD behaviors of all
  policies are deprecated and that a policy should be set to OLD only under
  specific short-term circumstances.  Projects should be ported to the NEW
  behavior and not rely on setting a policy to OLD.

-- INFO: will use GNU extensions for STDIO (useful for compressed I/O) since it seems accepted by your c++ compiler.
-- INFO: We prefer to use Readline than Editline.
Feedback welcome on OSX Readline ...
Use fuzzy detection for PLplot lib. (e.g. in /usr/lib)
-- Warning, if you have plplot version > 5.11 and wxWidgets enabled, please read carefully file README
-- Using a plplot library without private functions - workarounds will be used.
-- INFO: We prefer to use GraphicsMagick than ImageMagick
-- WARNING: Dynamic drivers may have to be disabled in plplot.
To make plplot work with ImageMagick:
use -DENABLE_DYNDRIVERS=OFF cmake option when compiling plplot or
use -DMAGICK=OFF here to not use ImageMagick in GDL.
-- HDF5: Using hdf5 compiler wrapper to determine C configuration
-- Summary

GDL - GNU DATA LANGUAGE [Python Module]
System                 Linux-5.4.0-29-generic
Files generated        Unix Makefiles
GDL output             
Installation prefix    /usr/local
C++ compiler           /usr/bin/c++ -O3 -DNDEBUG

-- Options
OpenMP support         ON (flag: -fopenmp)
WxWidgets              ON (libs:-L/usr/lib/x86_64-linux-gnu;-pthread;-lwx_baseu-3.0;-lwx_gtk3u_core-3.0;-lwx_gtk3u_adv-3.0; headers:)
ImageMagick            OFF
TIFF                   ON (libs:/usr/lib/x86_64-linux-gnu/libtiff.so; headers:/usr/include/x86_64-linux-gnu)
GeoTIFF                OFF
NetCDF                 ON (libs:netcdf; headers:)
HDF4                   ON (libs:/usr/lib/libmfhdfalt.so;/usr/lib/libdfalt.so;z;/usr/lib/x86_64-linux-gnu/libjpeg.so; headers:/usr/include/hdf)
HDF5                   ON (libs:/home/centarsirius/anaconda3/lib/libhdf5.so;/usr/lib/x86_64-linux-gnu/librt.so;/usr/lib/x86_64-linux-gnu/libpthread.so;/home/centarsirius/anaconda3/lib/libz.so;/usr/lib/x86_64-linux-gnu/libdl.so;/usr/lib/x86_64-linux-gnu/libm.so; headers:/home/centarsirius/anaconda3/include)
FFTW                   ON (libs:/usr/lib/x86_64-linux-gnu/libfftw3.so;/usr/lib/x86_64-linux-gnu/libfftw3f.so; headers:/usr/include)
Libproj4               ON (libs:/usr/lib/x86_64-linux-gnu/libproj.so; headers:/usr/include)
MPI                    ON (libs:/usr/lib/x86_64-linux-gnu/openmpi/lib/libmpi_cxx.so;/usr/lib/x86_64-linux-gnu/openmpi/lib/libmpi.so; headers:)
Python                 ON (libs:/usr/lib/x86_64-linux-gnu/libpython3.8.so; headers:/usr/include/python3.8)
UDUNITS-2              ON (libs:/usr/lib/x86_64-linux-gnu/libudunits2.so; headers:/usr/include)
EIGEN3                 ON (libs:; headers:/usr/include/eigen3)
GRAPHICSMAGICK         ON (libs:/usr/lib/libGraphicsMagick.so;/usr/lib/libGraphicsMagick++.so; headers:/usr/include/GraphicsMagick)
GRIB                   OFF
GLPK                   OFF
SHAPELIB               ON (libs:/usr/lib/x86_64-linux-gnu/libshp.so; headers:/usr/include)
EXPAT                  ON (libs:/usr/lib/x86_64-linux-gnu/libexpat.so; headers:/usr/include)
pslib                  ON (libs:/usr/lib/x86_64-linux-gnu/libps.so; headers:/usr/include)
Xlib                   ON (libs:/usr/lib/x86_64-linux-gnu/libSM.so;/usr/lib/x86_64-linux-gnu/libICE.so;/usr/lib/x86_64-linux-gnu/libX11.so;/usr/lib/x86_64-linux-gnu/libXext.so; headers:/usr/include)
libpng                 ON (libs:; headers:)

-- Mandatory modules
Plplot                 ON (libs:/usr/lib/x86_64-linux-gnu/libplplot.so;/usr/lib/x86_64-linux-gnu/libplplotcxx.so; headers:/usr/include)
GNU Readline           ON (libs:/usr/lib/x86_64-linux-gnu/libreadline.so;/usr/lib/x86_64-linux-gnu/libhistory.so; headers:/usr/include)
BSD Editline           OFF
GSL                    ON (libs:/usr/lib/x86_64-linux-gnu/libgsl.so;/usr/lib/x86_64-linux-gnu/libgslcblas.so; headers:/usr/include)
Zlib                   ON (libs:/usr/lib/x86_64-linux-gnu/libz.so; headers:/usr/include)
RPC                    ON (libs:; headers:/usr/include)
(N)curses              ON (libs:/usr/lib/x86_64-linux-gnu/libncurses.so;/usr/lib/x86_64-linux-gnu/libform.so; headers:/usr/include)

-- Configuring done
CMake Warning at src/CMakeLists.txt:196 (add_library):
  Cannot generate a safe runtime search path for target gdl because files in
  some directories may conflict with libraries in implicit directories:

    runtime library [libncurses.so] in /usr/lib/x86_64-linux-gnu may be hidden by files in:
      /home/centarsirius/anaconda3/lib
    runtime library [libform.so.6] in /usr/lib/x86_64-linux-gnu may be hidden by files in:
      /home/centarsirius/anaconda3/lib
    runtime library [libreadline.so.8] in /usr/lib/x86_64-linux-gnu may be hidden by files in:
      /home/centarsirius/anaconda3/lib
    runtime library [libhistory.so.8] in /usr/lib/x86_64-linux-gnu may be hidden by files in:
      /home/centarsirius/anaconda3/lib
    runtime library [libz.so.1] in /usr/lib/x86_64-linux-gnu may be hidden by files in:
      /home/centarsirius/anaconda3/lib
    runtime library [libpng16.so.16] in /usr/lib/x86_64-linux-gnu may be hidden by files in:
      /home/centarsirius/anaconda3/lib
    runtime library [libgslcblas.so.0] in /usr/lib/x86_64-linux-gnu may be hidden by files in:
      /home/centarsirius/anaconda3/lib
    runtime library [libtiff.so.5] in /usr/lib/x86_64-linux-gnu may be hidden by files in:
      /home/centarsirius/anaconda3/lib
    runtime library [libfftw3.so.3] in /usr/lib/x86_64-linux-gnu may be hidden by files in:
      /home/centarsirius/anaconda3/lib
    runtime library [libfftw3f.so.3] in /usr/lib/x86_64-linux-gnu may be hidden by files in:
      /home/centarsirius/anaconda3/lib
    runtime library [libexpat.so.1] in /usr/lib/x86_64-linux-gnu may be hidden by files in:
      /home/centarsirius/anaconda3/lib
    runtime library [libSM.so.6] in /usr/lib/x86_64-linux-gnu may be hidden by files in:
      /home/centarsirius/anaconda3/lib
    runtime library [libICE.so.6] in /usr/lib/x86_64-linux-gnu may be hidden by files in:
      /home/centarsirius/anaconda3/lib
    runtime library [libX11.so.6] in /usr/lib/x86_64-linux-gnu may be hidden by files in:
      /home/centarsirius/anaconda3/lib
    runtime library [libXext.so.6] in /usr/lib/x86_64-linux-gnu may be hidden by files in:
      /home/centarsirius/anaconda3/lib

  Some of these libraries may not be found correctly.

-- Generating done
-- Build files have been written to: /home/centarsirius/gdl/build
centarsirius@sachin-Legion-Y540:~/gdl/build$ make check
make: *** No rule to make target 'check'.  Stop.

now i am back to only 1 issue, more of a warning, the version mismatch of libtiff

Thank you all @slayoo @alaingdl @mvandam @GillesDuvert for your quick replies and support

[slayoo]

make check will not work if you are compiling as a Python module. The Python module testsuite can be run with python3 -m pytest ../testsuite/python/test-GDL.py

[GillesDuvert]

thank you @centarsirius !

[centarsirius]

So apart from the make check, I can use very thing else normally, right? And dual way support for python would be deafulty invoked or should I enable them through some code when I want to do cross-platform work?

[slayoo]

You should be able to do "make check" (and it's highly recommended) when compiling standalone gdl (i.e., -DPYTHON_MODULE=OFF).

In the case of Python module (-DPYTHON_MODULE=ON), python3 -m pytest ../testsuite/python/test-GDL.py will run the Python module tests.

Please report if that works.

[centarsirius]

hmm, looks like this is complicated. Anaconda python is 3.7 and it doesn't go well with GDL, so i had to disable that one, and now GDL is intalled only in python 3.8 but i still get this

centarsirius@sachin-Legion-Y540:~/gdl/build$ python3 -m pytest ../testsuite/python/test-GDL.py
============================= test session starts ==============================
platform linux -- Python 3.8.2, pytest-5.4.2, py-1.8.1, pluggy-0.13.1
rootdir: /home/centarsirius/gdl
collected 0 items / 1 error                                                    

==================================== ERRORS ====================================
________________ ERROR collecting testsuite/python/test-GDL.py _________________
ImportError while importing test module '/home/centarsirius/gdl/testsuite/python/test-GDL.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
../testsuite/python/test-GDL.py:12: in <module>
    import GDL
E   ImportError: libhdf5.so.103: cannot open shared object file: No such file or directory
=========================== short test summary info ============================
ERROR ../testsuite/python/test-GDL.py
!!!!!!!!!!!!!!!!!!!! Interrupted: 1 error during collection !!!!!!!!!!!!!!!!!!!!
=============================== 1 error in 0.13s ===============================

edit: i tried searching for the error, seems to be mostly a problem with opencv, tried almost all solution like reinstalling h5py, libhdf-dev, libhdf-10 etc, none worked tried to manually check and i got this

centarsirius@sachin-Legion-Y540:~$ python3.8
Python 3.8.2 (default, Apr 27 2020, 15:53:34) 
[GCC 9.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import gdl
>>> GDL.pro(fib,10)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
NameError: name 'GDL' is not defined

[centarsirius]

I think i'm just gonna go back to GDL without python support as i can't call GDL from my main work env (conda) anyway, and it can also cause complications with a lot of unwanted libraries being installed which i have to manually find and uninstall one after the other.

[GillesDuvert]

If I walk back this whole #752 issue, we started with a simple question about GDL examples and procedures on stock Ubuntu and finished on a very specialized treatment of gdl-to-python obscure problems in the case of self compilation of GDL. If I may draw some conclusions:

• Issues in GDL get quick attention from the authors :+1:
• Counsels almost always finish in a suggestion to recompile GDL, as there is always an immense gap between the version present in the ditstributions and the last development one :-1:
• Default GDL compilation, counsels and advertizement MUST NOT mention the existence of the Python bridge, that is actually an undocumented unmaintained and full of issues, stub.

and

• for clarity, issues should have only one subject (in this case it was "where are the procedures" (or so)) and any other business should be kept separately (in this case the difficulty to run a Python-aware GDL with Conda). Nobody will expect an issue called "Missing some inbuilt .pro files .." to contain useful information about Python bridging with GDL.

[centarsirius]

Yes, i was actually gonna bring this up, that there are three unrelated issues in side this issue thread. And calling python from GDL still has better documentation but calling GDL from python has negligible development and documentation as specified in python.txt itself

2. Calling GDL from python
==========================

NOTE: Due to the neglectible feedback, the python module
development is currently on hold.
If you use it, you are urged to let us know.

Most people who use GDL are still using age old compiles (like my professor) so we must look to it that the package in ubuntu repository is always the latest or at most 1 subversion behind

[slayoo]

Re Ubuntu packaging, the problem is also with the fact that new package versions do not target older Ubuntu distributions. Even if we update a package, someone using older Ubuntu version does not have an easy access to it. For instance, currently the default Travis Linux environment uses Ubuntu Xenial where the newest GDL is ... 0.9.6 (dating back to 2015)!

Re Python bridge, let me somehow oppose - thanks to @opoplawski and @olebole, we recently moved to Python 3 and we have a reasonable test coverage for the Python module, which is tested not only here on Travis but also as a part of Debian workflow. So it's not entirely true that it is unmaintained or "on hold"...

One of my conclusions from this issue is that our CMake infrastructure is still not handling well less standard environments (e.g., the numpy issue) and is not providing useful messages. One way to improve the situation would likely be to shift from hand-written .cmake files for detecting things like numpy to more standard solutions available in newer CMake versions (e.g., https://cmake.org/cmake/help/latest/module/FindPython3.html that has an option to detect NumPy). Same for GSL and other packages (see #417)

[centarsirius]

The warnings i was getting due to libtiff are now gone. i deleted all the previous compiles, recompiled plplot, and GDL without python support, just plain GDL. So the original issue of this thread is solved and the warnings too, so i think we should open a new thread for the python error and close this one.

[alaingdl]

Ten years ago when we started to use CMake with my student Maxime, we had very few Find*.cmake files around and Maxime created most of them (10+), from scratch or templates.

Yes it is time to cleanup !

But it is time consuming to make regressions of various systems (I maintain more than 15 VMs just to check as frequently as possible two important cases : compiling with most options, and with very few options)

I would say it would have been a very topic for a student this year, but it is quite impossible to start an internship this year here due to the situation.