search

brycefrank / pyfor

Tools for analyzing aerial point clouds of forest data.
MIT License
93 stars 19 forks source link

Silent failure of cloud class? Laz support? #24

Closed bw4sz closed 5 years ago

bw4sz commented 5 years ago

Hey Bryce, does pyfor still support .laz?

I just git cloned, and made a new conda env

MacBook-Pro:pyfor ben$ conda activate pyfor_env
(pyfor_env) MacBook-Pro:pyfor ben$ python
Python 3.6.6 | packaged by conda-forge | (default, Jul 26 2018, 09:55:02)
[GCC 4.2.1 Compatible Apple LLVM 6.1.0 (clang-602.0.53)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import pyfor
>>> pc=pyfor.cloud.Cloud("/Users/ben/Documents/DeepLidar/data/SJER/SJER_002.laz")
>>> pc

pc.data Traceback (most recent call last): File "", line 1, in AttributeError: 'Cloud' object has no attribute 'data'

https://www.dropbox.com/s/i1dnfuvl5y4x06m/SJER_003.laz?dl=0

I've tried with a few files.

It all looks good in R. I think its well formatted.

> a<-readLAS("/Users/ben/Dropbox/Weecology/NEON/SJER_003.laz")
> a
class        : LAS (LASF v1.3)
memory       : 889.8 Kb 
extent       : 257388 , 257428 , 4111280 , 4111320 (xmin, xmax, ymin, ymax)
area         : 1596.612 m² (convex hull)
points       : 11241 points,  NA pulses
density      : 7.04 points/m²,  NA pulses/m²
field names  : X Y Z gpstime Intensity ReturnNumber NumberOfReturns ScanDirectionFlag EdgeOfFlightline Classification Synthetic_flag Keypoint_flag Withheld_flag ScanAngle UserData PointSourceID 
coord. ref.  : NA
brycefrank commented 5 years ago

Try this. I have not added lastools to the environment.yml. Install and then re-run.

conda install -c conda-forge lastools -n pyfor_env

Even if this does fix it, .laz is not supported in a sensical way. Unfortunately, laspy uses a very slow procedure to extract the .laz information. It is something I am looking into.

bw4sz commented 5 years ago

Thanks for quick response.

Not quite there.

(pyfor_env) MacBook-Pro:pyfor ben$ conda install -c conda-forge lastools
Solving environment: done

==> WARNING: A newer version of conda exists. <==
  current version: 4.5.4
  latest version: 4.5.11

Please update conda by running

    $ conda update -n base conda

## Package Plan ##

  environment location: /Users/ben/miniconda3/envs/pyfor_env

  added / updated specs:
    - lastools

The following packages will be downloaded:

    package                    |            build
    ---------------------------|-----------------
    blas-1.0                   |              mkl           5 KB
    numpy-1.15.4               |   py36h6a91979_0          35 KB
    mkl_fft-1.0.6              |           py36_0         159 KB  conda-forge
    scikit-learn-0.20.0        |   py36h4f467ca_1         5.4 MB
    lastools-20171231          |       hfc679d8_0         668 KB  conda-forge
    mkl-2019.0                 |              118       154.4 MB
    intel-openmp-2019.0        |              118         1.0 MB
    mkl_random-1.0.2           |           py36_0         377 KB  conda-forge
    scipy-1.1.0                |   py36h28f7352_1        15.4 MB
    numpy-base-1.15.4          |   py36h8a80b8c_0         4.1 MB
    ------------------------------------------------------------
                                           Total:       181.6 MB

The following NEW packages will be INSTALLED:

    intel-openmp: 2019.0-118
    lastools:     20171231-hfc679d8_0                   conda-forge
    mkl:          2019.0-118
    mkl_fft:      1.0.6-py36_0                          conda-forge
    mkl_random:   1.0.2-py36_0                          conda-forge
    numpy-base:   1.15.4-py36h8a80b8c_0

The following packages will be UPDATED:

    libgfortran:  3.0.0-1                               conda-forge --> 3.0.1-h93005f0_2
    numpy:        1.15.4-py36_blas_openblashb06ca3d_0   conda-forge [blas_openblas] --> 1.15.4-py36h6a91979_0

The following packages will be DOWNGRADED:

    blas:         1.1-openblas                          conda-forge --> 1.0-mkl
    scikit-learn: 0.20.0-py36_blas_openblash00c3548_201 conda-forge [blas_openblas] --> 0.20.0-py36h4f467ca_1
    scipy:        1.1.0-py36_blas_openblashb06ca3d_202  conda-forge [blas_openblas] --> 1.1.0-py36h28f7352_1

Proceed ([y]/n)? y

Downloading and Extracting Packages
blas-1.0             |    5 KB | ################################################################################################################################ | 100%
numpy-1.15.4         |   35 KB | ################################################################################################################################ | 100%
mkl_fft-1.0.6        |  159 KB | ################################################################################################################################ | 100%
scikit-learn-0.20.0  |  5.4 MB | ################################################################################################################################ | 100%
lastools-20171231    |  668 KB | ################################################################################################################################ | 100%
mkl-2019.0           | 154.4 MB | ############################################################################################################################### | 100%
intel-openmp-2019.0  |  1.0 MB | ################################################################################################################################ | 100%
mkl_random-1.0.2     |  377 KB | ################################################################################################################################ | 100%
scipy-1.1.0          | 15.4 MB | ################################################################################################################################ | 100%
numpy-base-1.15.4    |  4.1 MB | ################################################################################################################################ | 100%
Preparing transaction: done
Verifying transaction: done
Executing transaction: done
(pyfor_env) MacBook-Pro:pyfor ben$ python
Python 3.6.6 | packaged by conda-forge | (default, Jul 26 2018, 09:55:02)
[GCC 4.2.1 Compatible Apple LLVM 6.1.0 (clang-602.0.53)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import pyfor
>>> pc=pyfor.cloud.Cloud("/Users/ben/Documents/DeepLidar/data/SJER/SJER_002.laz")
>>> print(pc)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/ben/Documents/pyfor/pyfor/cloud.py", line 122, in __str__
    min =  [float('{0:.2f}'.format(elem)) for elem in self.data.min]
AttributeError: 'Cloud' object has no attribute 'data'
>>>
brycefrank commented 5 years ago

Alright then! I will dig into this in a few hours. I have a suspicion what needs fixing. I will push the hotfix to master sometime today.

brycefrank commented 5 years ago

Ben,

The issue is sort-of fixed over on the laz_fix branch. This fix allows a file extension to be '.laz' and still go through the importing process. I was able to get the newly added test.laz in the testing suite to load and display points. I have not searched for other situations quite yet that may also break. But I think for the most part you should be good if you do the following:

(in your pyfor directory)

git fetch
git checkout laz_fix
pip install . --upgrade

This is assuming you already had a working installation of laszip.

I am having some trouble getting some new tests to pass on travis, there seems to be some weird issue with the path and conda and laspy. It may all be entertwined. Give the above a shot and let me know if it is still broken.

brycefrank commented 5 years ago

Another update, I switched from using laszip to lastools on conda, this seemed to do the trick for travis. I am going to leave that fix there for a day or two and ensure nothing else breaks.

bw4sz commented 5 years ago

Okay, working through this now. I've got a laszip version on the comp, but its probably not in path. If it s a dependency, maybe add that as a note? I'll check that in a sec.

(pyfor_env) MacBook-Pro:pyfor ben$ git status
On branch laz_fix
Your branch is up-to-date with 'origin/laz_fix'.
nothing to commit, working tree clean
(pyfor_env) MacBook-Pro:pyfor ben$ python
Python 3.6.6 | packaged by conda-forge | (default, Jul 26 2018, 09:55:02)
[GCC 4.2.1 Compatible Apple LLVM 6.1.0 (clang-602.0.53)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import pyfor
/Users/ben/miniconda3/envs/pyfor_env/lib/python3.6/site-packages/sklearn/externals/joblib/externals/cloudpickle/cloudpickle.py:47: DeprecationWarning:

the imp module is deprecated in favour of importlib; see the module's documentation for alternative uses

>>> pc=pyfor.cloud.Cloud("/Users/ben/Documents/DeepLidar/data/SJER/SJER_002.laz")
>>> print(pc)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/ben/Documents/pyfor/pyfor/cloud.py", line 126, in __str__
    return(out)
UnboundLocalError: local variable 'out' referenced before assignment
bw4sz commented 5 years ago

I take it back, laszip is in path. I wasn't sure how conda was going to play with brew install. Looks fine to me.

(pyfor_env) MacBook-Pro:pyfor ben$ which laszip
/Users/ben/miniconda3/envs/pyfor_env/bin/laszip
bw4sz commented 5 years ago

Just to confirm its not that file.

(pyfor_env) MacBook-Pro:pyfor ben$ ls
CHANGELOG.md        MANIFEST        docs            pyfor           readthedocs.yml     requirements.txt    setup.cfg
License.txt     README.md       environment.yml     pyfortest       requirements.rtd.txt    samples         setup.py
(pyfor_env) MacBook-Pro:pyfor ben$ cd pyfortest/
(pyfor_env) MacBook-Pro:pyfortest ben$ ls
__init__.py data        performance.py  test_pyfor.py
(pyfor_env) MacBook-Pro:pyfortest ben$ cd data/
(pyfor_env) MacBook-Pro:data ben$ ls
chm.tif     clip.dbf    clip.prj    clip.qpj    clip.shp    clip.shx    test.las    test.laz
(pyfor_env) MacBook-Pro:data ben$ python
Python 3.6.6 | packaged by conda-forge | (default, Jul 26 2018, 09:55:02)
[GCC 4.2.1 Compatible Apple LLVM 6.1.0 (clang-602.0.53)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import pyfor
/Users/ben/miniconda3/envs/pyfor_env/lib/python3.6/site-packages/sklearn/externals/joblib/externals/cloudpickle/cloudpickle.py:47: DeprecationWarning:

the imp module is deprecated in favour of importlib; see the module's documentation for alternative uses

>>> pc=pyfor.cloud.Cloud("test.laz")
>>> print(pc)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/ben/miniconda3/envs/pyfor_env/lib/python3.6/site-packages/pyfor/cloud.py", line 126, in __str__
    return(out)
UnboundLocalError: local variable 'out' referenced before assignment
brycefrank commented 5 years ago

This is turning out to be a tricky one, Ben. I see you have had some trouble with laspy / laszip as well. I wonder if it may be related.

There are a couple of details worth noting, just for your information and for me to see later:

  1. When I run unit tests in PyCharm I run into your original error. The test fails on https://github.com/brycefrank/pyfor/blob/0d45e189bae492a93cd5299a3cba01165ad2f87a/pyfortest/test_pyfor.py#L73

  2. However, when I run unit tests from the console the error does not occur. I am able to load the .laz file with no error and all tests pass.

  3. The latest build has passed with no errors on Travis. So I expect something is wonky with my local installation of the environment. I am also not testing the print statement that you are using in your above examples. I will add that to the unit tests and fix the tests.

  4. I need to restructure the unit tests so that they cleanly test all of the three supported file types .ply, .las and .laz for all methods/attributes of Cloud. I'll make this objective secondary. Once that is completed and working I will merge laz_fix to master. If all goes well this should be done today.

1 vs. 2 are a total mystery to me. I will try uninstalling lastools/laszip completely from my environment and otherwise and see if that might help.

I have a chunk of the day I was going to commit to pyfor already, so I will tackle this beast first in a couple of hours or so.

bw4sz commented 5 years ago

Okay, i'll dig into it too. But your sense is that this an environment issue. I was just following the README, i'll start trying to isolate what's happening. I was running from shell, so too much shouldn't be declared.

brycefrank commented 5 years ago

I can tell you already the problem is here (you can also see I thought this would become an issue earlier on line 117!)

https://github.com/brycefrank/pyfor/blob/0d45e189bae492a93cd5299a3cba01165ad2f87a/pyfor/cloud.py#L118-L120

Something you can do now to alleviate your problem (tell me if anything else breaks) is to just override what the extension is. This may effect some things down the line when you want to write back to .laz which is not something I am totally sure laspy even supports. I will have to check.

pc = ("your_laz.laz")
pc.extension = '.las'

I really need to spend time building out handling multiple file types. I think I will just make the ad-hox fixes for you today and then spend a week or two building it properly on 0.3.2. I plan on adding more file type support so it is best to do this in a robust way.

bw4sz commented 5 years ago

Confirmed that the workaround is fine. I'll let you know if I run into anything else.

brycefrank commented 5 years ago

Ben,

Also to note:

https://github.com/laspy/laspy/issues/73#issuecomment-419919558

I am learning a lot of the internals of laspy right now to implement spatial indexing. There is a lot of work to be done with laspy I think with regard to .laz. It is a severe limitation and I may end up trying to get that sorted out over there in the next year. It will take time.

brycefrank commented 5 years ago

Last thing Ben.

It was a messy fight. I forgot to pull updates to my local repository yesterday! Aye-yai-yai....

I have added the new tests to check for .laz loading. Turns out my other issues were problems with pycharm. I installed a global version of lastools as a workaround for now. It is concerning. Something to do with how pycharm searches for the path that is different from the Linux terminal. A problem for another day.

The tests are rudimentary at best, but they passed on my local machine. I will ensure they do so on travis before merging to master.

brycefrank commented 5 years ago

Closing for now. If anything the remaining PyCharm issue is a problem in laspy

Sajjadmanal commented 5 years ago

@bw4sz @brycefrank How can I read .laz file usinf laspy? I am getting below error: LaspyException: Error mapping file: Laszip was not found on the system