search

USF-IMARS / wv-land-cover

:earth_americas: Processing scripts for decision-tree land use classification on worldview 2 imagery
5 stars 5 forks source link

Syntax errors while running pgc_ortho.py #23

Open iparask opened 4 years ago

iparask commented 4 years ago

Hello,

I'm trying to run pgc_ortho.py on PSC Bridges, and I am getting a series of syntax errors. For example:

(GulfCoastEnv) [paraskev@r005 GulfCoast]$ python pgc_ortho.py -p 4326 -c ns -t UInt16 -f GTiff --no-pyramids $INPUT_DIR ./test_output/
  File "pgc_ortho.py", line 57
    except RuntimeError, e:
                       ^
SyntaxError: invalid syntax

after fixing this, I got:

(GulfCoastEnv) [paraskev@r005 GulfCoast]$ python pgc_ortho.py -p 4326 -c ns -t UInt16 -f GTiff --no-pyramids $INPUT_DIR ./test_output/
Traceback (most recent call last):
  File "pgc_ortho.py", line 10, in <module>
    from lib import ortho_utils as ortho_utils
  File "/pylon5/mc3bggp/paraskev/GulfCoast/lib/ortho_utils.py", line 74
    srs = osr.SpatialReference()
                               ^
TabError: inconsistent use of tabs and spaces in indentation

I'm running the code on a Linux system:

LSB Version:    :core-4.1-amd64:core-4.1-noarch:cxx-4.1-amd64:cxx-4.1-noarch:desktop-4.1-amd64:desktop-4.1-noarch:languages-4.1-amd64:languages-4.1-noarch:printing-4.1-amd64:printing-4.1-noarch
Distributor ID: CentOS
Description:    CentOS Linux release 7.6.1810 (Core)
Release:    7.6.1810
Codename:   Core

using Anaconda Python 3.7 from conda-forge.

bspitzbart commented 4 years ago

We just had an ICEBERG CI meeting and the team felt this issue would be best looked into on the USF-IMARS end so that we do not inadvertently break anything else.  Is the code originally from PGC?  Maybe you can reach out to them?

7yl4r commented 4 years ago

Thanks for the bug report. These errors are because the code from PGC we are using is older and written for python2. The rest of the code here should be run with python3.

I have just added pgc's imagery_utils as a git submodule of this project:

git submodule add -b v1.5.2 https://github.com/PolarGeospatialCenter/imagery_utils.git wv_classify/pgc_imagery_utils

We should try using their code directly instead of the duplicated copy in this repo. I will add more detailed instructions and some more tests in the near future, but for now you can update the latest submodule with git submodule update --init --recursive --remote and execute the python files therein directly.

I will remove our copies of the imagery_utils files living at this project root once I am able to determine and document which version they are.

7yl4r commented 4 years ago

Here is the PGC/imagery_utils code repo. Better instructions for using pgc_ortho.py can be found there.

iparask commented 4 years ago

Hello @7yl4r, I managed to get it going, but I am still getting errors. Please excuse the questions, since I am not a polar scientist, and some may seem just as request for documentation.

I am getting this error now:

(GulfCoastEnv) [paraskev@r005 GulfCoastUtils]$ mkdir test_output
(GulfCoastEnv) [paraskev@r005 GulfCoastUtils]$ python pgc_ortho.py -p 4326 -c ns -t UInt16 -f GTiff --no-pyramids $INPUT_DIR ./test_output/
12-09-2019 16:04:39 INFO- Number of src images: 2
12-09-2019 16:04:39 INFO- Number of incomplete tasks: 2
12-09-2019 16:04:39 INFO- Submitting Tasks
12-09-2019 16:04:39 INFO- Image: 16FEB12162517-M1BS-057380245010_01_P001.NTF
12-09-2019 16:04:39 INFO- Working Dir: /pylon5/mc3bggp/paraskev/GulfCoastUtils/test_output
Traceback (most recent call last):
  File "pgc_ortho.py", line 285, in <module>
    main()
  File "pgc_ortho.py", line 268, in main
    results[task.name] = task.method(src, dstfp, task_arg_obj)
  File "/pylon5/mc3bggp/paraskev/GulfCoastUtils/lib/ortho_functions.py", line 400, in process_image
    info, rc = GetImageStats(args, info, target_extent_geom)
  File "/pylon5/mc3bggp/paraskev/GulfCoastUtils/lib/ortho_functions.py", line 752, in GetImageStats
    vendor, sat = utils.get_sensor(info.srcfn)
  File "/pylon5/mc3bggp/paraskev/GulfCoastUtils/lib/utils.py", line 107, in get_sensor
    return vendor, sat.upper()
AttributeError: 'NoneType' object has no attribute 'upper'
(GulfCoastEnv) [paraskev@r005 GulfCoastUtils]$

Am I missing something as an input or should I contact PGC directly?

7yl4r commented 4 years ago

Please open an issue for this with PGC/imagery_utils here https://github.com/PolarGeospatialCenter/imagery_utils/issues. It is important that we go through the motions to document everything on github so that PGC stays in the loop. Please also include the version of imagery_utils you are using and the value of $INPUT_DIR in your environment (ie the output of echo $INPUT_DIR)

I think this might be a simple file not found error, but imagery_utils should throw a better error if that is the case.

On Mon, Dec 9, 2019 at 4:08 PM Ioannis Paraskevakos < notifications@github.com> wrote:

Hello @7yl4r https://github.com/7yl4r, I managed to get it going, but I am still getting errors. Please excuse the questions, since I am not a polar scientist, and some may seem just as request for documentation.

I am getting this error now:

(GulfCoastEnv) [paraskev@r005 GulfCoastUtils]$ mkdir test_output (GulfCoastEnv) [paraskev@r005 GulfCoastUtils]$ python pgc_ortho.py -p 4326 -c ns -t UInt16 -f GTiff --no-pyramids $INPUT_DIR ./test_output/ 12-09-2019 16:04:39 INFO- Number of src images: 2 12-09-2019 16:04:39 INFO- Number of incomplete tasks: 2 12-09-2019 16:04:39 INFO- Submitting Tasks 12-09-2019 16:04:39 INFO- Image: 16FEB12162517-M1BS-057380245010_01_P001.NTF 12-09-2019 16:04:39 INFO- Working Dir: /pylon5/mc3bggp/paraskev/GulfCoastUtils/test_output Traceback (most recent call last): File "pgc_ortho.py", line 285, in main() File "pgc_ortho.py", line 268, in main results[task.name] = task.method(src, dstfp, task_arg_obj) File "/pylon5/mc3bggp/paraskev/GulfCoastUtils/lib/ortho_functions.py", line 400, in process_image info, rc = GetImageStats(args, info, target_extent_geom) File "/pylon5/mc3bggp/paraskev/GulfCoastUtils/lib/ortho_functions.py", line 752, in GetImageStats vendor, sat = utils.get_sensor(info.srcfn) File "/pylon5/mc3bggp/paraskev/GulfCoastUtils/lib/utils.py", line 107, in get_sensor return vendor, sat.upper() AttributeError: 'NoneType' object has no attribute 'upper' (GulfCoastEnv) [paraskev@r005 GulfCoastUtils]$

Am I missing something as an input or should I contact PGC directly?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/USF-IMARS/wv2-processing/issues/23?email_source=notifications&email_token=AAIAV7QM63UAVYWV3MTJ6R3QX2XWDA5CNFSM4JH5AY22YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEGKWSFQ#issuecomment-563439894, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAIAV7QVRVBMFKP44K3YMBLQX2XWDANCNFSM4JH5AY2Q .

--

===========================================

Tylar Murray http://tylar.info, Ph.D.

IMaRS http://imars.marine.usf.edu/ Research Systems & Software Engineer

USF CMS http://marine.usf.edu - KRC 3119-B

------------------------------------------

schedule : g-calendar

https://calendar.google.com/calendar?cid=NWRuOHRubTBmczlmZjN0cTVhMGczbnBqbXNAZ3JvdXAuY2FsZW5kYXIuZ29vZ2xlLmNvbQ, youcanbookme https://7yl4r.youcanbook.me/

===========================================

7yl4r commented 4 years ago

@mjm8 can you provide the python and gdal versions that the USF supercomputer is using? Hopefully the same versions will be available on Bridges.

mjm8 commented 4 years ago

Hmm how do I determine which version it calls? I don't specify in the submit script.

On Fri, Jan 10, 2020 at 3:31 PM Tylar notifications@github.com wrote:

@mjm8 https://github.com/mjm8 can you provide the python and gdal versions that the USF supercomputer is using? Hopefully the same versions will be available on Bridges.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/USF-IMARS/wv2-processing/issues/23?email_source=notifications&email_token=AH7I4KFJKHFIPRWUSXXYVGDQ5DLLLA5CNFSM4JH5AY22YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEIVD7TY#issuecomment-573194191, or unsubscribe https://github.com/notifications/unsubscribe-auth/AH7I4KEUKHSBTXSCN6KPLPLQ5DLLLANCNFSM4JH5AY2Q .

-- Matt McCarthy, Ph.D. Biological Oceanography College of Marine Science University of South Florida 140 7th Avenue South, KRC 3117 St Petersburg, FL 33701-5016 727-553-1186

7yl4r commented 4 years ago

add to your slurm script:

python --version
gdal-config --version

You should then be able to find it in the output log for the job

mjm8 commented 4 years ago

Okay, it's submitted now. Just have to wait for it to run and spit out the log then I'll let you know.

Have a nice weekend, Matt

On Fri, Jan 10, 2020 at 4:02 PM Tylar notifications@github.com wrote:

add to your slurm script:

python --version gdal-config --version

You should then be able to find it in the output log for the job

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/USF-IMARS/wv2-processing/issues/23?email_source=notifications&email_token=AH7I4KGQKPZZT2AVNJLFKGLQ5DO7ZA5CNFSM4JH5AY22YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEIVGOOY#issuecomment-573204283, or unsubscribe https://github.com/notifications/unsubscribe-auth/AH7I4KHV7FIUMSKXTDYNXHDQ5DO7ZANCNFSM4JH5AY2Q .

-- Matt McCarthy, Ph.D. Biological Oceanography College of Marine Science University of South Florida 140 7th Avenue South, KRC 3117 St Petersburg, FL 33701-5016 727-553-1186

mjm8 commented 4 years ago

Python 2.7.5 Gdal 1.11.4

Matt McCarthy, Ph.D.

Postdoctoral Research Scholar College of Marine Science University of South Florida 140 7th Avenue South, KRC 3117 St Petersburg, FL 33701-5016 727-553-1186

On Fri, Jan 10, 2020, 4:08 PM Matt McCarthy mjm8@mail.usf.edu wrote:

Okay, it's submitted now. Just have to wait for it to run and spit out the log then I'll let you know.

Have a nice weekend, Matt

On Fri, Jan 10, 2020 at 4:02 PM Tylar notifications@github.com wrote:

add to your slurm script:

python --version gdal-config --version

You should then be able to find it in the output log for the job

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/USF-IMARS/wv2-processing/issues/23?email_source=notifications&email_token=AH7I4KGQKPZZT2AVNJLFKGLQ5DO7ZA5CNFSM4JH5AY22YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEIVGOOY#issuecomment-573204283, or unsubscribe https://github.com/notifications/unsubscribe-auth/AH7I4KHV7FIUMSKXTDYNXHDQ5DO7ZANCNFSM4JH5AY2Q .

-- Matt McCarthy, Ph.D. Biological Oceanography College of Marine Science University of South Florida 140 7th Avenue South, KRC 3117 St Petersburg, FL 33701-5016 727-553-1186

iparask commented 4 years ago

I was able to workaround this issue on Bridges. Here is what I did:

[paraskev@r125 ~]$  module load python2/2.7.11_gcc_np1.11
[paraskev@r125 ~]$ virtualenv myenv
[paraskev@r125 ~]$ source myenv/bin/activate
(myenv)[paraskev@r125 ~]$export PYTHONPATH=<path_to_env>/myenv/lib/python2.7/site-packages:$PYTHONPATH
(myenv)[paraskev@r125 ~]$pip install pygdal==1.11.4.6
(myenv)[paraskev@r125 ~]$export PYTHONPATH=<path_to_env>/myenv/lib/python2.7/site-packages/osgeo/:$PYTHONPATH

This allowed my successfully execute the PGC code.