Installing coastseg on Win10

[csherwood-usgs]

ERROR: Could not install packages due to an OSError: [WinError 5] Access is denied: 'C:\\Users\\csherwood\\AppData\\Local\\Temp\\1\\pip-uninstall-9e60dyo4\\jsonschema.exe'
Consider using the `--user` option or check the permissions.

I get this error when trying to run the pip install command.

[2320sharon]

Hi Chris, Thanks for reporting this error. Its hard to tell from the error message alone what the problem might be. Can you walk through the steps you took before you encountered this error?

Thank you!

[csherwood-usgs]

I followed the steps outlined in the README.md. I don't get any SSL errors. I usually have no problem doing installs, esp. with conda. Pip is sometimes more of a problem. I don't have a pip .ini, but it is not obvious where I should put it....there is a pip folder in ..\Appdata\local, which is not one of the choices on the search path...and there is something in the cache there from today. I was using the mamba solver...I disabled that, but it did not seem to make any difference.

I think it would probably be most efficient to chat and screen share.

[2320sharon]

Thank you Chris, I think you're onto something when you mention there's something stuck in cache. The error message mentions the -user flag so maybe pip install coastseg --user would solve this issue. I'm happy to have a video call with you tomorrow or any time that works better for you. In the meantime I'll double check the location I have the pip.ini file stored and update the wiki to make it easier to locate.

Thank you for your feedback and help.

[dbuscombe-usgs]

Sorry I'm late to the party here. Yes, like Sharon said, I think you may benefit from some housekeeping. I tend to do this regularly:

conda update -n base conda
conda clean --all
pip install --upgrade pip

then try pip install -U coastseg --user. The -U flag will make sure you are installing the latest version. mamba tends only to help conda resolve dependencies quickly, so that's a red herring here.

I don't have a pip .ini, but it is not obvious where I should put it....there is a pip folder in ..\Appdata\local, which is not one of the choices on the search path.

We have previously verified that storing your pip.ini file in any location provided by pip config -v list works (this is the list of places pip will look for the .ini file). So, if your pip.ini file is in one of those locations, I would suggest it is probably the path to the certificate file that is the problem (assuming you are on secure network?) So please double check the paths to your DOIRootCA2.cer file (and if you don't have one, get one from your IT person)

[http]
    sslCAInfo = C:\Users\username\Documents\DOIRootCA2.cer
[global] 
    cert = C:\Users\username\Documents\DOIRootCA2.cer

We have TCA stuff in the morning but could help you troubleshoot after 12pm pacific

[2320sharon]

I started back at the beginning and I've found a good solution that I'm probably going to put in wiki because its easier. Let me know what you think @dbuscombe-usgs I verified these commands work on the network and worked when I deleted all my pip.ini files. pip install --user --cert <path to cert file> <pkg_name> Ex: pip install --user --cert c:\users\sfitzpatrick\Documents\DOIRootCA2.cer coastseg

Benefits

• No need to create a pip.ini file
• easier to understand

Downsides

• user have to specify the --cert flag everytime they need to pip install

[2320sharon]

Okay I found a much easier way of setting the pip.ini file in 3 easy steps. I bet it will fix this issue. Here is the single command you will need: all you need to is replace the path to my cert file with the path to your cert file and pip should work properly

pip config set global.cert C:\Users\sfitzpatrick\Documents\DOIRootCA2.cer

I updated the wiki with this new method https://github.com/Doodleverse/CoastSeg/wiki/How-to-Install-CoastSeg-with-pip-on-a-secure-network

[dbuscombe-usgs]

Nice work!! Hopefully this fixes it for you, Chris!

[csherwood-usgs]

That is great...but I don't have that DOI cert file on either my TIC laptop or my WH workstation. The former is a locked-down machine that works on the VPN, the latter is on our Woods Hole "science network" that piggybacks on the WHOI network, and is not completely locked down. I have admin accounts on both machines. Does the file need to exist?

[2320sharon]

Thanks for describing your two setups having that context helps narrow down the possible cause of the issue. That being said its hard to know exactly what the issue is. If the issue is an SSL error aka your network doesn't allow traffic without the cert, then the pip config solution would work. However, in your case it appears the issue you're having is a permissions issue. I'm guessing you already tried the pip install coastseg --user command but it didn't work.

If using the -user flag to install pip packages doesn't work, then it might be worth clearing your cache because something there might be causing the issue. You You mentioned before in your previous email that you were running these commands in Admin mode on anaconda prompt, so you should have access to modify files.

Let's verify a few things to see if the issue is something else

1. Run pip --version
2. Run pip list
   • This will test if pip is functioning properly
3. Run pip install requests --user
   • This will test if pip can download a basic package like requests using the user permissions

[dbuscombe-usgs]

It's also likely that WH is different, somehow not requiring the .cer file, but something else. Kathy and Julia are already using coastseg in WH, so they may be able to offer some insight

[csherwood-usgs]

conda create --name coastseg python=3.10 -y

[This command is no longer in your installation instructions...should it be? At this point, I have pip installed in the environment.]

(coastseg) PS D:\crs\src\CoastSeg> pip --version
pip 23.2.1 from C:\Users\csherwood\Anaconda3\envs\coastseg\lib\site-packages\pip (python 3.10)
(coastseg) PS D:\crs\src\CoastSeg> pip list
Package    Version
---------- -------
pip        23.2.1
setuptools 68.0.0
wheel      0.41.1pi

pip install requests --user
...
Installing collected packages: urllib3, idna, charset-normalizer, certifi, requests
  WARNING: The script normalizer.exe is installed in 'C:\Users\csherwood\AppData\Roaming\Python\Python310\Scripts' which is not on PATH.
  Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
Successfully installed certifi-2023.7.22 charset-normalizer-3.2.0 idna-3.4 requests-2.31.0 urllib3-2.0.4

conda install -c conda-forge geopandas jupyterlab -y

[That seemed to work...no errors, and jupyter lab runs at this point.]

Then I did pip install coastset --user

and got a lot of warnings like the first one, plus:

  Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.  

ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.  

jupyter-events 0.7.0 requires jsonschema[format-nongpl]>=4.18.0, but you have jsonschema 4.17.3 which is incompatible.  

Successfully installed PyQt5-Qt5-5.15.2 PyQt5-sip-12.12.2 PyWavelets-1.4.1 absl-py-1.4.0 aiohttp-3.8.5 aiosignal-1.3.1 area-1.1.1 astropy-5.3.2 astunparse-1.6.3 async-timeout-4.0.3 bqplot-0.12.40 cachetools-5.3.1 chardet-5.2.0 cloudpickle-2.2.1 coastsat-package-0.1.13 coastseg-0.0.73 colour-0.1.5 dask-2023.8.0 doodleverse-utils-0.0.35 earthengine-api-0.1.364 filelock-3.12.2 flatbuffers-23.5.26 frozenlist-1.4.0 fsspec-2023.6.0 gast-0.4.0 gdown-4.7.1 geojson-3.0.1 google-api-core-2.11.1 google-api-python-client-2.97.0 google-auth-2.22.0 google-auth-httplib2-0.1.0 google-auth-oauthlib-1.0.0 google-cloud-core-2.3.3 google-cloud-storage-2.10.0 google-crc32c-1.5.0 google-pasta-0.2.0 google-resumable-media-2.5.0 googleapis-common-protos-1.60.0 grpcio-1.57.0 h5py-3.9.0 httplib2-0.22.0 huggingface-hub-0.16.4 imageio-2.31.1 ipyevents-2.0.1 ipyfilechooser-0.6.0 ipyleaflet-0.17.3 ipytree-0.2.2 ipywidgets-8.1.0 jsonschema-4.17.3 jupyterlab-widgets-3.0.8 keras-2.13.1 lazy_loader-0.3 leafmap-0.23.2 libclang-16.0.6 locket-1.0.0 markdown-3.4.4 multidict-6.0.4 numpy-1.24.3 oauthlib-3.2.2 opt-einsum-3.3.0 partd-1.4.0 protobuf-4.24.0 pyasn1-0.5.0 pyasn1-modules-0.3.0 pyerfa-2.0.0.3 pyqt5-5.15.9 pyrsistent-0.19.3 pyshp-2.3.1 pystac-1.8.3 pystac-client-0.7.2 python-box-7.0.1 regex-2023.8.8 requests-oauthlib-1.3.1 rsa-4.9 safetensors-0.3.2 scikit-image-0.21.0 scooby-0.7.2 tensorboard-2.13.0 tensorboard-data-server-0.7.1 tensorflow-2.13.0 tensorflow-estimator-2.13.0 tensorflow-intel-2.13.0 tensorflow-io-gcs-filesystem-0.31.0 termcolor-2.3.0 tifffile-2023.8.12 tokenizers-0.13.3 toolz-0.12.0 tqdm-4.66.1 traittypes-0.2.1 transformers-4.31.0 typing-extensions-4.5.0 uritemplate-4.1.1 urllib3-1.26.16 versioneer-0.29 werkzeug-2.3.7 whitebox-2.3.1 whiteboxgui-2.3.0 widgetsnbextension-4.0.8 wrapt-1.15.0 yarl-1.9.2

But after that jupyter lab no longer works:

Traceback (most recent call last):
  File "C:\Users\csherwood\Anaconda3\envs\coastseg\Scripts\jupyter-lab-script.py", line 5, in <module>
    from jupyterlab.labapp import main
  File "C:\Users\csherwood\Anaconda3\envs\coastseg\lib\site-packages\jupyterlab\__init__.py", line 8, in <module>
    from .handlers.announcements import (
  File "C:\Users\csherwood\Anaconda3\envs\coastseg\lib\site-packages\jupyterlab\handlers\announcements.py", line 14, in <module>
    from jupyter_server.base.handlers import APIHandler
  File "C:\Users\csherwood\Anaconda3\envs\coastseg\lib\site-packages\jupyter_server\base\handlers.py", line 23, in <module>
    from jupyter_events import EventLogger
  File "C:\Users\csherwood\Anaconda3\envs\coastseg\lib\site-packages\jupyter_events\__init__.py", line 3, in <module>
    from .logger import EVENTS_METADATA_VERSION, EventLogger
  File "C:\Users\csherwood\Anaconda3\envs\coastseg\lib\site-packages\jupyter_events\logger.py", line 18, in <module>
    from .schema import SchemaType
  File "C:\Users\csherwood\Anaconda3\envs\coastseg\lib\site-packages\jupyter_events\schema.py", line 18, in <module>
    from .validators import draft7_format_checker, validate_schema
  File "C:\Users\csherwood\Anaconda3\envs\coastseg\lib\site-packages\jupyter_events\validators.py", line 41, in <module>
    JUPYTER_EVENTS_SCHEMA_VALIDATOR = Draft7Validator(  # type: ignore
TypeError: create.<locals>.Validator.__init__() got an unexpected keyword argument 'registry'

[csherwood-usgs]

Is it that jsonschema error? How do I install the proper version?

I tried this, but no changes were made:

(coastseg) PS D:\crs\src\CoastSeg> conda update jsonschema
Error while loading conda entry point: conda-build (argument of type 'NoneType' is not iterable)
Error while loading conda entry point: conda-build (argument of type 'NoneType' is not iterable)
Collecting package metadata (current_repodata.json): done
Solving environment: |
Warning: 2 possible package resolutions (only showing differing packages):
  - conda-forge/noarch::pyparsing-3.0.9-pyhd8ed1ab_0, conda-forge/win-64::matplotlib-base-3.7.2-py310h51140c5_0
  - conda-forge/noarch::pyparsing-3.1.1-pyhd8ed1ab_0, conda-forge/win-64::matplotlib-base-3.7.1-py310h51140c5done

# All requested packages already installed.

Got the same result with:
conda install jsonschema>=4.18.0

[csherwood-usgs]

OK...finally - did a pip uninstall jsonschema then pip install jsonschema==4.18.0.

[Forgot to use the --user flag.]

Got this:

Installing collected packages: jsonschema
  Attempting uninstall: jsonschema
    Found existing installation: jsonschema 4.19.0
    Uninstalling jsonschema-4.19.0:
      Successfully uninstalled jsonschema-4.19.0
ERROR: Could not install packages due to an OSError: [WinError 5] Access is denied: 'C:\\Users\\csherwood\\AppData\\Local\\Temp\\1\\pip-uninstall-w4uxbldz\\jsonschema.exe'
Consider using the `--user` option or check the permissions.

(coastseg) PS D:\crs\src\CoastSeg> conda list | grep json*
Error while loading conda entry point: conda-build (argument of type 'NoneType' is not iterable)
Error while loading conda entry point: conda-build (argument of type 'NoneType' is not iterable)
json5                     0.9.14             pyhd8ed1ab_0    conda-forge
jsonpointer               2.0                        py_0    conda-forge
jsonschema                4.18.0                   pypi_0    pypi
jsonschema-specifications 2023.7.1           pyhd8ed1ab_0    conda-forge
jsonschema-with-format-nongpl 4.19.0             pyhd8ed1ab_1    conda-forge
python-fastjsonschema     2.18.0             pyhd8ed1ab_0    conda-forge
python-json-logger        2.0.7              pyhd8ed1ab_0    conda-forge

So, despite an apparent error, it looks like I have the version I need.

And jupyter lab works!

I think I am done.

[dbuscombe-usgs]

Great! Your tenacity paid off. You did encounter some unusual problems, though, so I will use your experience to try to update the instructions. For example, we'll add the conda create --name coastseg python=3.10 -y back to the conda recipe, and I'll look more into what jsonschema is and whether that needs to be a specific version.

For now, we are recommending that you stick with the SDS_coastsat_classifier workflow. We are close to having the deep-learning approach working, but have a couple of features to add and bugs to fix there.

You may be interested in attending Sharon's next office hours, where she helps usgs coastseg users with issues, discuss bugs and new features. I will let you know when the next one is (within the next 1-2 weeks)

[csherwood-usgs]

Thanks to both of you for the help. The --user thing was key. Of course, my GEE account has been disabled, so I had to apply for one with my personal account. But my real interest is in finding Planet imagery and then segmenting it to look at unvegetated areas. Is there any reason Planet can't be added to the pipeline?

[dbuscombe-usgs]

Planet is not a priority yet until we fully explore the other image sources, so is a planned feature for V2. V2 is slated for later this year. I agree it will be really nice to have it, and we're going to make it happen soon I hope!

V1 is what we're working towards now (we are close), which will have its own validation/benchmarking. Planet imagery is slated for V2 (as I detail here), along with a bunch of other potential improvements, but in the meantime you can use the CoastSat-Planet toolbox for now https://github.com/ydoherty/CoastSat.PlanetScope for shoreline extraction.

Soon, I will update the roadmap in the wiki to reflect this, where I plan to outline the phases of planned coastseg development.

Things other than shoreline extraction, such as vegetation detection, will likely be in a future Coastseg version (like, V3) or perhaps might require a slightly different tool, but existing Zoo models are available to you to try once you have your vegetation. Also note that I could connect you with others in the USGS who are already using Planet imagery for mapping vegetation - they are a little ahead of us in terms of developing workflows.

[csherwood-usgs]

Ok...it seems like I am on my own....or working with the others that are looking at Planet and veg...is that Nicholas et al.? I am not very interested in shorelines, and I don't think Landsat has enough resolution...not sure about Sentinel 2. So I am interested in tools for identifying satellite images, and then probably using a Zoo model or other approach for veg. I am willing to be a tester for any Planet stuff.

[csherwood-usgs]

Note that, after I got mine installed, Filipe put together a coastseg installation on conda-forge. I have not tried it. He was interested in knowing how the CoastSat model was archived.