search

Esri / deep-learning-frameworks

Installation support for Deep Learning Frameworks for the ArcGIS System
448 stars 110 forks source link

Update for RTX 30 Series GPUs (CUDA 11) #17

Closed oSilentCartographer closed 2 years ago

oSilentCartographer commented 3 years ago

When can we expect updated libraries for RTX 30 series GPUs? They have been out for over 7 months now. I have an RTX 3090 but it cannot run any deep learning tasks through ArcGIS Pro. My understanding is that these new GPUs require CUDA 11 support. It looks like these libraries need updated versions of PyTorch and TensorFlow as well.

knoopum commented 3 years ago

I'm looking forward to these updates as well. Thanks!

Hulseyj commented 3 years ago

New RTX 3060. I am experiencing an error when I use "Detect Objects Using Deep Learning" in ArcGIS Pro with new RTX 3060. The model used is usa_building_footprints.dlpk. I am using ArcGIS Pro 2.7.3. I have no issues running the model with my GTX 1650 Super GPU, same computer, same settings.

I have a post on this issue in the ESRI Community forums here: https://community.esri.com/t5/arcgis-pro-questions/detect-objects-using-deep-learning-error-with-new-rtx-3060/m-p/1050015#M40037

Still trying to find a resolution.

oSilentCartographer commented 3 years ago

Sorry I clicked close accidentally!

I meant to reply to Hulseyj:

I have communicated with ESRI support multiple times about this issue and have yet to get any details about future plans to upgrade the libraries for the latest GPUs. They did acknowledge that this is caused by lack of CUDA 11 support.

njpamperin commented 3 years ago

Using an NVIDIA RTX 3080 laptop (and CUDA 11.3) and having issues with deep learning tools as well (device side CUDA assert errors). Just adding my name to the list of those having issues with the new 30xx series cards. Hoping for a solution soon.

Samsa95 commented 3 years ago

I'm using a RTX 3060 laptop version and got problems with the tools as well (nan values when using the GPU), probably because of the above mentioned lack of actual libraries. Hope for a fast update to use this tools.

njpamperin commented 3 years ago

I'm using a RTX 3060 laptop version and got problems with the tools as well (nan values when using the GPU), probably because of the above mentioned lack of actual libraries. Hope for a fast update to use this tools.

Yes, depending on the model I am trying to train on the RTX 3080 (laptop) I also get nan for training and validation loss values when trying to use GPU processing. This mainly happens with SSD whereas trying to train MaskRCNN produces the device side CUDA assert error.

MattBC90 commented 3 years ago

Oh my god, guys! I have been trying to figure out why the heck I couldn't get any of this working as it should for over a month now. I have a 3080 and I've encountered pretty much all of the same errors mentioned.

I got around some of the ArcGIS notebook issues for model training by doing it inside of an external Jupyter browser, but just as soon as I was making progress, I got stonewalled when I kept getting "nan" values while trying to train the thing. I thought I was going crazy!

I am actually exploring various means to do GIS Deep Learning so I guess I have little choice but to look at how to do these steps through open source code, which ArcGIS seems to use anyway. More of a hassle but it's either this way or no project.

PetrieSwart1970 commented 3 years ago

Hi All. Hving the same problem with ArcGIS PRO 2.7 and RTX 3060. CUDA version installed is 11.2.

njpamperin commented 3 years ago

Oh my god, guys! I have been trying to figure out why the heck I couldn't get any of this working as it should for over a month now. I have a 3080 and I've encountered pretty much all of the same errors mentioned.

I got around some of the ArcGIS notebook issues for model training by doing it inside of an external Jupyter browser, but just as soon as I was making progress, I got stonewalled when I kept getting "nan" values while trying to train the thing. I thought I was going crazy!

I am actually exploring various means to do GIS Deep Learning so I guess I have little choice but to look at how to do these steps through open source code, which ArcGIS seems to use anyway. More of a hassle but it's either this way or no project.

Make sure you check the Esri community post referenced by Hulseyj above. It sounds like Esri is working on a solution.

MattBC90 commented 3 years ago

Make sure you check the Esri community post referenced by Hulseyj above. It sounds like Esri is working on a solution.

Either way, I doubt anything will be working by the time I have to complete my project. Maybe they just assumed everybody with RTX 3000 series GPUs were only using them for mining, which is only partially true.

Just for the record, issues I've run into with my RTX 3080:

scdub commented 3 years ago

Thank you for providing details on your workflows and your interest in using the deep learning tools on Ampere based GPUs. As you have discovered, Ampere cards are not consistently supported by CUDA 10 based solutions. The deep learning collection includes eight separate GPU backed packages, which are all compiled against the CUDA Toolkit 10.1 release. Unfortunately, some of these dependencies have not been fully updated to support Ampere currently, and CUDA 10.1 does not natively support the binary formats Ampere provides, only CUDA 11.1+ does as you can see in this table:

image

When you run GPU based code, there are two primary execution paths: a targeted cubin binary which is optimized for the specific architecture, and is what you generally want to use. The fallback is PTX, which will JIT compile a kernel from an older GPU target onto the current GPU. However, this translation in places can be problematic and introduces errors as has been documented above, though it will work in many cases with simpler CUDA code (the last column provides details on what binary and PTX versions are included in our bundled PyTorch build).

Future Pro plans

Our plan for Pro 2.8 is to continue distributing a collection of installers and package set targeted at CUDA Tookilt 10.1, and works for GPUs of the last Kepler generation (3.7 with Tesla K80), Maxwell, Pascal, Turing, and Volta GPUs. This collection of packages has gone through extensive validation, and covers the GPUs for a broad base of our users, and has upstream support from the packages we are redistributing or building.

For Ampere users, we have been working toward a technology preview that will be available by UC 2021, which will have some known limitations but allow key arcgis.learn, DL based geoprocessing tools and core libraries to function, using CUDA 11 and cubins for the CC 8.6 platform. There won't be an installer for this technology preview, but it will be available as a conda installable metapackage, and once we post this, we will be sure to respond to this issue as well.

For the next release of Pro which will arrive Q4 2021, we will support Ampere out of the box as a target for the deep learning installers, and will have a solution based on CUDA Toolkit 11.

SrikantaESV commented 3 years ago

I'm also facing the same problem, any lead on this would be highly appreciated-

Traceback (most recent call last): File "c:\program files\arcgis\pro\Resources\ArcToolbox\toolboxes\Image Analyst Tools.tbx\TrainDeepLearningModel.tool\tool.script.execute.py", line 297, in execute() File "c:\program files\arcgis\pro\Resources\ArcToolbox\toolboxes\Image Analyst Tools.tbx\TrainDeepLearningModel.tool\tool.script.execute.py", line 271, in execute callbacks=[ProgressCallback(training_model_object, model_type, max_epochs, out_folder)] File "C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\Lib\site-packages\arcgis\learn\models_arcgis_model.py", line 798, in fit self.learn.fit_one_cycle(epochs, lr, callbacks=callbacks, kwargs) File "C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\Lib\site-packages\fastai\train.py", line 23, in fit_one_cycle learn.fit(cyc_len, max_lr, wd=wd, callbacks=callbacks) File "C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\Lib\site-packages\fastai\basic_train.py", line 200, in fit fit(epochs, self, metrics=self.metrics, callbacks=self.callbacks+callbacks) File "C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\Lib\site-packages\fastai\basic_train.py", line 101, in fit loss = loss_batch(learn.model, xb, yb, learn.loss_func, learn.opt, cb_handler) File "C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\Lib\site-packages\fastai\basic_train.py", line 26, in loss_batch out = model(xb) File "C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\Lib\site-packages\torch\nn\modules\module.py", line 532, in call result = self.forward(input, kwargs) File "C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\Lib\site-packages\torchvision\models\detection\generalized_rcnn.py", line 70, in forward proposals, proposal_losses = self.rpn(images, features, targets) File "C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\Lib\site-packages\torch\nn\modules\module.py", line 532, in call result = self.forward(*input, **kwargs) File "C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\Lib\site-packages\torchvision\models\detection\rpn.py", line 472, in forward boxes, scores = self.filter_proposals(proposals, objectness, images.image_sizes, num_anchors_per_level) File "C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\Lib\site-packages\torchvision\models\detection\rpn.py", line 392, in filter_proposals keep = box_ops.remove_small_boxes(boxes, self.min_size) File "C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\Lib\site-packages\torchvision\ops\boxes.py", line 95, in remove_small_boxes keep = keep.nonzero().squeeze(1) RuntimeError: copy_if failed to synchronize: cudaErrorAssert: device-side assert triggered

oSilentCartographer commented 3 years ago

Do we have any more info about the "technology preview" that was slated to be ready by UC 2021?

Hornbydd commented 3 years ago

I was advised to add to this list to emphasise that my attempts to do pixel classification using deep learning have failed possibly due to the fact I have an RTX 3070 card. My attempts and failure has been documented here:

https://community.esri.com/t5/arcgis-pro-questions/help-with-deep-learning-and-pixel-classification/m-p/1081251#M43557

I do hope the appropriate libraries are updated soon!

geoli91 commented 3 years ago

I'm looking forward to these updates. Also, I'd like to know if we can upgrade the versionf of PyTorch and CUDA ourselves via conda or other means? I tried to do that, but conda gave me a bunch of conflict.

oSilentCartographer commented 3 years ago

I spent a lot of time trying to figure out if I could manually update PyTorch, TensorFlow, etc., but it all ended in failure. I think I got some things installed by changing the version limits. But there are many internal references/dependencies regarding the versions -- e.g., this thing will only work if PyTorch is version ___. I tried editing those too, but still could not get things working.

We are still waiting for the technology preview that might help some of us circumvent these issues. It is overdue at this point; hoping they release it soon.

pchristensengis commented 3 years ago

conda esri deep-learning-ampere-preview is live on anaconda.org

https://anaconda.org/Esri/deep-learning-ampere-preview

cleebp commented 3 years ago

Howdy all 🤠

We greatly appreciate your patience waiting for this work, as mentioned by @pchristensengis the deep-learning-ampere-preview is now available on the esri conda channel here.

System requirements:

  1. Pro 2.8 should be installed, ideally with the latest update patches installed (Pro >=2.8.2 should work)
    • If installing on server you need to be on version 10.9.0
  2. If you have previously installed the Deep Learning setups for Pro 2.8 you'll need to uninstall the Deep Learning Libraries installation via the msi installer.

Install steps:

  1. Launch the Python Command Prompt via Start > ArcGIS > Python Command Prompt
  2. Create a clone with conda create --clone arcgispro-py3 --name dl-env-name --pinned
    • Note the --pinned flag which is essential for Pro clones to work within Pro itself
  3. Activate your clone with activate dl-env-name
  4. Run conda install -c esri deep-learning-ampere-preview
    • Note this may take some time to solve, also initial failed solves are expected (it should eventually pass after relaxing the solver)
  5. Run proswap dl-env-name to swap Pro's active environment to your deep learning clone
  6. Congrats, you should be able to use deep learning features/tools inside or outside the application now when using this environment!

It has been difficult for us to test this work with limited access to the latest graphic cards series, if you run into any issues please report them here, thanks again for your patience!

Hornbydd commented 3 years ago

Hi this is great news and I'm willing to be a Guinea pig for testing on an RTX 3070 card. But as a true Guinea pig I tend to shy away from the install steps you have described above and prefer to use the msi installer as I can't mess it up! I understand I need to run the add & remove tool to un-install the deep learning libraries but can you provide a new msi installer that have your latest updates? Or am i asking for too much and it's not ready for being distributed as an msi installer, because that was nice and simple.

oSilentCartographer commented 3 years ago

It looks like some of us are running into an Error 002667 when trying to use the detect objects tool -- the error shows up when first trying to load the dlpk.

More details here: https://community.esri.com/t5/arcgis-pro-questions/detect-objects-using-deep-learning-error-with-new/m-p/1092551#M44833

cleebp commented 3 years ago

More details here: https://community.esri.com/t5/arcgis-pro-questions/detect-objects-using-deep-learning-error-with-new/m-p/1092551#M44833

I'm following up on this with our ampere machine and am having trouble reproducing, if the --pinned flag was omitted during the cloning step that would cause an error like this.

@oSilentCartographer could you provide what your ampere environment reports after running conda info and conda list in the python command prompt?

oSilentCartographer commented 3 years ago

I did use the --pinned flag and tried a second time with a fresh install of ArcGIS Pro 2.8.2.

Here are the reports: active environment : Pro28-PreviewAmp active env location : C:\Program Files\ArcGIS\Pro\bin\Python\envs\Pro28-PreviewAmp user config file : C:\Users\x.condarc populated config files : C:\Program Files\ArcGIS\Pro\bin\Python.condarc conda version : 4.9.2 conda-build version : not installed python version : 3.7.10.final.0 virtual packages : cuda=11.4=0 win=0=0 __archspec=1=x86_64 base environment : C:\Program Files\ArcGIS\Pro\bin\Python (writable) channel URLs : https://conda.anaconda.org/esri/win-64 https://conda.anaconda.org/esri/noarch https://repo.anaconda.com/pkgs/main/win-64 https://repo.anaconda.com/pkgs/main/noarch https://repo.anaconda.com/pkgs/r/win-64 https://repo.anaconda.com/pkgs/r/noarch https://repo.anaconda.com/pkgs/msys2/win-64 https://repo.anaconda.com/pkgs/msys2/noarch package cache : C:\Users\x\AppData\Local\ESRI\conda\pkgs C:\Program Files\ArcGIS\Pro\bin\Python\pkgs C:\Users\x.conda\pkgs envs directories : C:\Program Files\ArcGIS\Pro\bin\Python\envs C:\Users\x\AppData\Local\ESRI\conda\envs C:\Users\x.conda\envs platform : win-64 user-agent : conda/4.9.2 requests/2.25.1 CPython/3.7.10 Windows/10 Windows/10.0.19041 administrator : True netrc file : None offline mode : False

packages in environment at C:\Program Files\ArcGIS\Pro\bin\Python\envs\Pro28-PreviewAmp:

Name Version Build Channel _tflow_select 2.3.0 gpu absl-py 0.13.0 py37haa95532_0 appdirs 1.4.4 py_0 arcgis 1.8.5 py37_1783 esri arcgispro 2.8 0 esri arcpy 2.8 py37_arcgispro_29734 [arcgispro] esri arrow-cpp 1.0.1 3 esri ase 3.19.1 py37_0 esri asn1crypto 1.4.0 py_0 astor 0.8.1 py37haa95532_0 astunparse 1.6.3 py_0 atomicwrites 1.4.0 py_0 attrs 21.2.0 pyhd3eb1b0_0 azure-core 1.12.0 py_0 esri azure-storage-blob 12.8.0 py_0 esri backcall 0.2.0 pyhd3eb1b0_0 beautifulsoup4 4.9.3 pyha847dfd_0 black 19.10b0 py_0 blas 1.0 mkl bleach 3.3.0 pyhd3eb1b0_0 blinker 1.4 py37haa95532_0 boost 1.73.0 py37haa95532_11 bottleneck 1.3.2 py37h2a96729_1 brotlipy 0.7.0 py37h2bbff1b_1003 cached-property 1.5.2 py_0 cachetools 4.2.2 pyhd3eb1b0_0 catalogue 1.0.0 py37_0 esri certifi 2021.5.30 py37haa95532_0 cffi 1.14.6 py37h2bbff1b_0 cftime 1.0.0b1 py37_0 esri chardet 4.0.0 py37haa95532_1003 click 8.0.1 pyhd3eb1b0_0 cloudpickle 1.6.0 py_0 colorama 0.4.4 pyhd3eb1b0_0 cppzmq 4.4.1 2 esri cpuonly 1.0 0 esri cryptography 3.3.1 py37_0 esri cudatoolkit 11.2.0 h608a323_8 esri cudnn 8.1.0.77 h3e0f4f4_0 esri cycler 0.10.0 py37_0 cymem 2.0.5 py37hd77b12b_0 cython 0.29.24 py37hd77b12b_0 cython-blis 0.4.1 py37_0 esri cytoolz 0.11.0 py37he774522_0 dask-core 2021.8.0 pyhd3eb1b0_0 decorator 5.0.9 pyhd3eb1b0_0 deep-learning-ampere-preview 2.8 arcgispro_4 [arcgispro] esri defusedxml 0.7.1 pyhd3eb1b0_0 despatch 0.1.0 py37_0 esri entrypoints 0.3 py37_0 et_xmlfile 1.0.1 py_1001 fastai 1.0.60 py37_0 esri fastcache 1.1.0 py37he774522_0 fastprogress 0.2.3 py37_0 esri fasttext 0.9.2 py37h74a9793_0 esri filelock 3.0.12 pyhd3eb1b0_1 flake8 3.9.0 pyhd3eb1b0_0 freetype 2.10.1 vc14_0 [vc14] esri fsspec 2021.7.0 pyhd3eb1b0_0 future 0.18.2 py37_0 esri gast 0.3.3 py_0 gdal 2.3.3 arcgispro_py37_16747 [arcgispro] esri google-auth 1.21.3 py_0 google-auth-oauthlib 0.4.4 pyhd3eb1b0_0 google-pasta 0.2.0 py_0 googledrivedownloader 0.4 py37_0 esri graphviz 2.38 hfd603c8_2 grpcio 1.35.0 py37hc60d5dd_0 h5py 2.10.0 py37_arcgispro_10 [arcgispro] esri html5lib 1.1 py_0 icc_rt 2019.0.5 arcgispro_0 [arcgispro] esri idna 2.10 pyhd3eb1b0_0 imageio 2.8.0 py37_0 esri importlib-metadata 3.10.0 py37haa95532_0 importlib_metadata 3.10.0 hd3eb1b0_0 iniconfig 1.1.1 pyhd3eb1b0_0 intel-openmp 2020.0 arcgispro_166 [arcgispro] esri ipykernel 5.1.1 py37_0 esri ipython 7.21.0 py37_0 esri ipython_genutils 0.2.0 pyhd3eb1b0_1 ipywidgets 7.4.2 py37_0 isodate 0.6.0 py_0 esri jdcal 1.4.1 py_0 jedi 0.18.0 py37_0 esri jinja2 2.11.3 pyhd3eb1b0_0 joblib 1.0.1 pyhd3eb1b0_0 jpeg 9d 0 esri json5 0.9.4 py37_0 esri jsonschema 3.2.0 py_2 jupyter_client 6.1.7 py_0 esri jupyter_console 6.2.0 py_2 esri jupyter_contrib_core 0.3.3 py37_3 esri jupyter_contrib_nbextensions 0.5.1 py37_10 esri jupyter_core 4.6.3 py37_2 esri jupyter_highlight_selected_word 0.2.0 py37_2 esri jupyter_latex_envs 1.4.4 py37_1 esri jupyter_nbextensions_configurator 0.4.1 py37_1 esri jupyterlab 2.2.7 py_0 esri jupyterlab_server 1.2.0 py_0 keepalive 0.5 py37_1 esri keras-applications 1.0.8 py_1 keras-preprocessing 1.1.2 pyhd3eb1b0_0 keyring 21.4.0 py37_0 esri kiwisolver 1.3.1 py37hd77b12b_0 laspy 1.7.0 py37_1 esri lerc 3.0 pyh39e3cac_0 esri libboost 1.73.0 h6c2663c_11 libopencv 4.5.0 py37_3 esri libpng 1.6.37 h2a8f88b_0 libprotobuf 3.17.2 h23ce68f_1 libsodium 1.0.18 1 esri libtiff 4.1.0 0 esri libuv 1.40.0 he774522_0 libwebp 1.2.0 h2bbff1b_0 libxml2 2.9.10 arcgispro_0 [arcgispro] esri libxslt 1.1.34 he774522_0 llvmlite 0.36.0 py37h34b8924_4 locket 0.2.1 py37haa95532_1 lxml 4.6.3 py37h9b66d53_0 lz4-c 1.9.3 h2bbff1b_1 markdown 3.3.4 py37haa95532_0 markupsafe 1.1.1 py37hfa6e2cd_1 matplotlib 3.3.1 py37_arcgispro_0 [arcgispro] esri mccabe 0.6.1 py37_1 mistune 0.8.4 py37hfa6e2cd_1001 mkl 2020.0 arcgispro_167 [arcgispro] esri mkl-service 2.3.0 py37h196d8e1_0 mkl_fft 1.3.0 py37h46781fe_0 mkl_random 1.2.0 py37_0 esri mpmath 1.2.1 py37haa95532_0 msrest 0.6.21 py_0 esri murmurhash 1.0.5 py37hd77b12b_0 mypy_extensions 0.4.3 py37_0 nbconvert 5.6.1 py37_0 esri nbformat 5.0.7 py_1 esri netcdf4 1.5.4 py37_arcgispro_6 [arcgispro] esri networkx 2.5 py37_0 esri ninja 1.10.2 h6d14046_1 nlohmann_json 3.7.0 1 esri nose 1.3.7 pyhd3eb1b0_1006 notebook 5.7.10 py37_0 ntlm-auth 1.4.0 py_0 esri numba 0.53.0 py37hf11a4ad_0 numexpr 2.7.3 py37hcbcaa1e_0 numpy 1.20.1 py37_0 esri numpy-base 1.20.1 py37_0 esri nvidia-ml-py3 7.352.0 py_0 esri oauthlib 3.1.1 pyhd3eb1b0_0 olefile 0.46 py37_0 onnx 1.9.0 py37_1 esri onnx-tf 1.8.0 py37_0 esri opencv 4.5.0 py37_3 esri openpyxl 3.0.7 pyhd3eb1b0_0 openssl 1.1.1i 0 esri opt_einsum 3.3.0 pyhd3eb1b0_1 packaging 21.0 pyhd3eb1b0_0 pandas 1.3.2 py37h6214cd6_0 pandocfilters 1.4.3 py37haa95532_1 parso 0.8.1 pyhd3eb1b0_0 partd 1.2.0 pyhd3eb1b0_0 pathspec 0.7.0 py_0 pefile 2019.4.18 py_0 pickleshare 0.7.5 pyhd3eb1b0_1003 pillow-simd 7.1.2 py37_3 esri pip 21.2.2 py37haa95532_0 plac 1.1.0 py37_1 plotly 4.5.4 py_0 esri pluggy 0.13.1 py37haa95532_0 pooch 1.0.0 py37_0 esri preshed 3.0.2 py37_0 esri pro_notebook_integration 2.8 py37_0 esri prometheus_client 0.8.0 py_0 esri prompt_toolkit 3.0.5 py_0 esri protobuf 3.17.2 py37hd77b12b_0 psutil 5.8.0 py37h2bbff1b_1 py 1.10.0 pyhd3eb1b0_0 py-boost 1.73.0 py37hac22706_11 py-opencv 4.5.0 py37_3 esri pyarrow 1.0.1 1 esri pyasn1 0.4.8 py_0 pyasn1-modules 0.2.8 py_0 pybind11 2.3.0 1 esri pycodestyle 2.6.0 pyhd3eb1b0_0 pycparser 2.20 py_2 pyflakes 2.2.0 pyhd3eb1b0_0 pygments 2.7.0 py_0 esri pyjwt 2.1.0 py37haa95532_0 pyopenssl 20.0.1 pyhd3eb1b0_1 pyparsing 2.4.7 pyhd3eb1b0_0 pypdf2 1.26.0 py_2 esri pyrsistent 0.17.3 py37he774522_0 pyshp 2.1.3 pyhd3eb1b0_0 pysocks 1.7.1 py37_1 pytest 6.1.1 py37_0 esri python 3.7.10 0 esri python-certifi-win32 1.2 py37_0 esri python-dateutil 2.8.2 pyhd3eb1b0_0 pytorch 1.4.0 py3.7_cuda11.2_cudnn8.1_1 esri pytz 2020.1 py37_0 esri pywavelets 1.1.1 py37he774522_2 pywin32-ctypes 0.2.0 pypi_0 pypi pywin32-security 228 py37_3 esri pywinpty 0.5.7 py37_0 esri pyyaml 5.4.1 py37h2bbff1b_1 pyzmq 19.0.2 py37_1 esri rdflib 5.0.0 py37_0 esri regex 2021.8.3 py37h2bbff1b_0 requests 2.25.1 pyhd3eb1b0_0 requests-kerberos 0.12.0 0 esri requests-negotiate-sspi 0.5.2 py37_1 esri requests-oauthlib 1.3.0 py_0 requests-toolbelt 0.9.1 py_0 requests_ntlm 1.1.0 py_0 esri retrying 1.3.3 py37_2 rsa 4.7.2 pyhd3eb1b0_1 sacremoses 0.0.43 py_0 esri saspy 3.6.1 py_0 esri scikit-image 0.17.2 py37_1 esri scikit-learn 0.23.2 py37h47e9c7a_0 scipy 1.6.2 py37h14eb087_0 send2trash 1.5.0 pyhd3eb1b0_1 sentencepiece 0.1.91 py37h74a9793_0 esri setuptools 52.0.0 py37haa95532_0 simplegeneric 0.8.1 py37_2 six 1.15.0 py_0 esri soupsieve 2.2.1 pyhd3eb1b0_0 spacy 2.2.4 py37_0 esri sparqlwrapper 1.8.5 py37_1 esri sqlite 3.36.0 h2bbff1b_0 srsly 1.0.2 py37_0 esri swat 1.8.1 py37_0 esri sympy 1.5.1 py37_0 esri tensorboard 2.4.0 pyhc547734_0 tensorboard-plugin-wit 1.6.0 py_0 tensorboardx 2.1 py_0 esri tensorflow 2.3.0 mkl_py37h10aaca4_0 tensorflow-addons 0.13.0 py37_2 esri tensorflow-base 2.3.0 eigen_py37h17acbac_0 tensorflow-estimator 2.5.0 pyh7b7c402_0 tensorflow-gpu 2.3.0 he13fc11_0 termcolor 1.1.0 py37haa95532_1 terminado 0.9.3 py37haa95532_0 testpath 0.4.4 pyhd3eb1b0_0 thinc 7.4.0 py37_0 esri threadpoolctl 2.2.0 pyhbf3da8f_0 tifffile 2020.10.1 py37h8c2d366_2 esri tokenizers 0.10.1 py37_0 esri toml 0.10.2 pyhd3eb1b0_0 toolz 0.11.1 pyhd3eb1b0_0 torch-cluster 1.5.4 py37_cuda11.2_1 esri torch-geometric 1.7.0 py37_1 esri torch-scatter 2.0.4 py37_cuda11.2_1 esri torch-sparse 0.6.1 py37_cuda11.2_1 esri torch-spline-conv 1.2.0 py37_cuda11.2_1 esri torchvision 0.5.0 py37_cpu [cpuonly] esri tornado 6.1 py37h2bbff1b_0 tqdm 4.62.1 pyhd3eb1b0_1 traitlets 4.3.3 py37_0 transformers 3.3.0 py_1 esri typed-ast 1.4.2 py37h2bbff1b_1 typeguard 2.7.0 py37_0 esri typing 3.10.0.0 py37haa95532_0 typing-extensions 3.10.0.0 hd3eb1b0_0 typing_extensions 3.10.0.0 pyh06a4308_0 ujson 4.0.2 py37hd77b12b_0 urllib3 1.26.6 pyhd3eb1b0_1 vc 14.2 h21ff451_1 vs2015_runtime 14.27.29016 h5e58377_2 wasabi 0.6.0 py37_0 esri wcwidth 0.2.5 py_0 webencodings 0.5.1 py37_1 werkzeug 1.0.1 pyhd3eb1b0_0 wheel 0.37.0 pyhd3eb1b0_0 widgetsnbextension 3.4.2 py37_0 win_inet_pton 1.1.0 py37_0 esri wincertstore 0.2 py37_0 winkerberos 0.7.0 py37_0 esri winpty 0.4.3 4 wrapt 1.12.1 py37he774522_1 x86cpu 0.4 py37_1 esri xarray 0.17.0 pyhd3eb1b0_0 xeus 0.24.1 1 esri xlrd 1.2.0 py37_0 xlwt 1.3.0 py37_0 xtl 0.6.5 1 esri xz 5.2.5 h62dcd97_0 yaml 0.2.5 0 esri zeromq 4.3.2 2 esri zipp 3.5.0 pyhd3eb1b0_0 zlib 1.2.11 h62dcd97_4 zstd 1.4.5 h04227a9_0

knoopum commented 3 years ago

Cannot get it to work either.

Tried several times to make sure I was reading the instructions correctly. This is on a fresh install of Pro 2.8.0, updated to 2.8.2, and then following the instructions, making sure --pinned is in there.

Something basic seems to be broken by deep-learning-ampere-preview, as even "from arcgis import GIS" will give you an error.

Amperecondainfo.txt Amperecondalist.txt

pchristensengis commented 3 years ago

On RTX 3070, I am also seeing the same 002667 in Detect Objects...

I tried using the downloaded usa_building_footprints.dlpk and the online Living Atlas version.

Info and List reports appear identical to others. Amperecondalist.txt Amperecondainfo.txt

Side Note: Something to try for anyone experiencing verification errors when installing deep-learning-ampere-preview.

I could not get the deep-learning-ampere-preview to install because I kept getting tensorflow-base verification errors like "...package is corrupt, ... could not find file..." however I could manually explore to the file's location and see the file was there.

It turns out that several of their files are buried a dozen folders deep and the combination of my long username and the packages being stored in %localappdata%, the file path was over the Windows 10 limit of 260 characters. I modified the windows filepath limits to allow paths that exceed 260 characters but that did not seem to affect conda.

I had to create a new location for packages closer to root in order to decrease the file path length and it finally worked.

oSilentCartographer commented 3 years ago

Anyone have luck getting this to work? Or any update on an additional fix?

(sorry I accidentally clicked "close" again)

aamrufl commented 3 years ago

I am also trying to train deep learning network after following the deep-learning-ampere-preview installation instructions on 3070 with no luck.

SrikantaESV commented 3 years ago

Is there any update or clue to fix the issue?

AbdullahAlfayez commented 3 years ago

We all have been waiting for a fix since March! Any update since you mentioned there will be a solution by November?

RossDF commented 2 years ago

Has there been any further development with this? I was under the impression that it was going to be part of the ArcGIS Pro 2.9 release, however I can't see any mention in patch notes.

data4point0 commented 2 years ago

Has there been any further development with this? I was under the impression that it was going to be part of the ArcGIS Pro 2.9 release, however I can't see any mention in patch notes.

I am also interested to see if there has been a 2.9 update in this regard - nothing mentioned in the patch

oSilentCartographer commented 2 years ago

I just installed 2.9 and the accompanying deep learning frameworks (follow conda install instructions), which do have updated versions of Pytorch, Tensorflow, etc.

But something still seems wrong. While my GPU is now being utilized with the Detect Objects tool, each time it ends in "ERROR 160722: The item was not found." Same outcome if I use the CPU too. I have never encountered this error when using this tool in the past (with older GPUs that I know to work).

oSilentCartographer commented 2 years ago

Upon more testing, it appears that having "Non Maximum Suppression" selected is causing the tool to fail. Unchecked and it works, but of course the output has overlapping detections that I do not want. Can anyone think of why this is happening?

AbdullahAlfayez commented 2 years ago

Upon more testing, it appears that having "Non Maximum Suppression" selected is causing the tool to fail. Unchecked and it works, but of course the output has overlapping detections that I do not want. Can anyone think of why this is happening?

Does training a model work with you though? @oSilentCartographer

oSilentCartographer commented 2 years ago

Upon more testing, it appears that having "Non Maximum Suppression" selected is causing the tool to fail. Unchecked and it works, but of course the output has overlapping detections that I do not want. Can anyone think of why this is happening?

Does training a model work with you though? @oSilentCartographer

I still have to test model training. But I did find that there is a separate standalone Non Maximum Suppression tool, and I can run it after Detect Objects (with NMS turned off), and that actually works. It's an extra step, but this is major progress!

In other words, only the NMS within the Detect Objects tool is broken, and it turns out that is more of a convenience than a necessity.

Hulseyj commented 2 years ago

I am happy to report that Detect Objects Using Deep Learning is running successfully with my RTX 3060 after performing a clean install of ArcPro 2.9.0 and the 2.9 Deep Learning Libraries. I am seeing a significant improvement in speed as compared to my RTX 1650 Super. I have tested the Building Footprint Extraction deep learning model that is published by ESRI on the Living Atlas. I see better results in terms of speed when I use the default tile size environment settings (I remove the model argument: tile size 512). I have run the model in ArcPro and Jupyter notebook successfully.

Thanks to all who have worked to solve the issue with using the deep learning tools on the new Ampere cards.

scdub commented 2 years ago

I believe the original issue raised here has been addressed with the DL installation for Pro 2.9, and Ampere cards are now part of our internal testing process. If anyone is having issues on Pro 2.9 using Ampere cards, let's move those conversations to new issues as this is now supported and is likely specific to particular tools or models.

leijiang1122 commented 1 year ago

Have you ever encountered 002667 error while performing deep learning target detection on arcgispro 3.0 with 3060 graphics card?