conda-forge / tensorflow-feedstock

A conda-smithy repository for tensorflow.
BSD 3-Clause "New" or "Revised" License
91 stars 81 forks source link

Parameter to MergeFrom() must be instance of same class: expected tensorflow.GraphDef got tensorflow.GraphDef. #364

Closed njzjz closed 5 months ago

njzjz commented 6 months ago

Solution to issue cannot be found in the documentation.

Issue

2023-12-23T02:13:08.8794601Z   File "/home/conda/feedstock_root/build_artifacts/deepmd-kit_1703296524930/_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_place/lib/python3.10/site-packages/deepmd/train/trainer.py", line 820, in save_checkpoint
2023-12-23T02:13:08.8796829Z     ckpt_prefix = self.saver.save(
2023-12-23T02:13:08.8798012Z   File "/home/conda/feedstock_root/build_artifacts/deepmd-kit_1703296524930/_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_place/lib/python3.10/site-packages/tensorflow/python/training/saver.py", line 1325, in save
2023-12-23T02:13:08.8804679Z     self.export_meta_graph(
2023-12-23T02:13:08.8805896Z   File "/home/conda/feedstock_root/build_artifacts/deepmd-kit_1703296524930/_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_place/lib/python3.10/site-packages/tensorflow/python/training/saver.py", line 1372, in export_meta_graph
2023-12-23T02:13:08.8806505Z     return export_meta_graph(
2023-12-23T02:13:08.8807259Z   File "/home/conda/feedstock_root/build_artifacts/deepmd-kit_1703296524930/_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_place/lib/python3.10/site-packages/tensorflow/python/training/saver.py", line 1709, in export_meta_graph
2023-12-23T02:13:08.8807691Z     meta_graph_def, _ = meta_graph.export_scoped_meta_graph(
2023-12-23T02:13:08.8808456Z   File "/home/conda/feedstock_root/build_artifacts/deepmd-kit_1703296524930/_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_place/lib/python3.10/site-packages/tensorflow/python/framework/meta_graph.py", line 1020, in export_scoped_meta_graph
2023-12-23T02:13:08.8819403Z     scoped_meta_graph_def = create_meta_graph_def(
2023-12-23T02:13:08.8820901Z   File "/home/conda/feedstock_root/build_artifacts/deepmd-kit_1703296524930/_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_place/lib/python3.10/site-packages/tensorflow/python/framework/meta_graph.py", line 557, in create_meta_graph_def
2023-12-23T02:13:08.8824777Z     meta_graph_def.graph_def.MergeFrom(graph_def)
2023-12-23T02:13:08.8825371Z TypeError: Parameter to MergeFrom() must be instance of same class: expected tensorflow.GraphDef got tensorflow.GraphDef.
2023-12-23T02:13:10.3584477Z WARNING: Tests failed for deepmd-kit-2.2.7-cuda112py310haa36818_4.conda - moving package to /home/conda/feedstock_root/build_artifacts/broken

I got this issue when rebuilding for TF 2.15: https://github.com/conda-forge/deepmd-kit-feedstock/pull/66

TF 2.14 or the pypi package don't have this issue, so this may be related to #361.

Installed packages

2023-12-23T02:12:08.0774096Z     _libgcc_mutex:            0.1-conda_forge                       conda-forge
2023-12-23T02:12:08.0774479Z     _openmp_mutex:            4.5-2_gnu                             conda-forge
2023-12-23T02:12:08.0774840Z     absl-py:                  2.0.0-pyhd8ed1ab_0                    conda-forge
2023-12-23T02:12:08.0775218Z     aiohttp:                  3.9.1-py310h2372a71_0                 conda-forge
2023-12-23T02:12:08.0775582Z     aiosignal:                1.3.1-pyhd8ed1ab_0                    conda-forge
2023-12-23T02:12:08.0775952Z     ase:                      3.22.1-pyhd8ed1ab_1                   conda-forge
2023-12-23T02:12:08.0776323Z     astunparse:               1.6.3-pyhd8ed1ab_0                    conda-forge
2023-12-23T02:12:08.0776684Z     async-timeout:            4.0.3-pyhd8ed1ab_0                    conda-forge
2023-12-23T02:12:08.0777276Z     attrs:                    23.1.0-pyh71513ae_1                   conda-forge
2023-12-23T02:12:08.0777663Z     binutils_impl_linux-64:   2.40-hf600244_0                       conda-forge
2023-12-23T02:12:08.0778029Z     binutils_linux-64:        2.40-hbdbef99_2                       conda-forge
2023-12-23T02:12:08.0778403Z     blinker:                  1.7.0-pyhd8ed1ab_0                    conda-forge
2023-12-23T02:12:08.0778772Z     blosc:                    1.21.5-h0f2a231_0                     conda-forge
2023-12-23T02:12:08.0779123Z     bracex:                   2.2.1-pyhd8ed1ab_0                    conda-forge
2023-12-23T02:12:08.0779487Z     brotli:                   1.1.0-hd590300_1                      conda-forge
2023-12-23T02:12:08.0779852Z     brotli-bin:               1.1.0-hd590300_1                      conda-forge
2023-12-23T02:12:08.0780210Z     brotli-python:            1.1.0-py310hc6cd4ac_1                 conda-forge
2023-12-23T02:12:08.0780583Z     bzip2:                    1.0.8-hd590300_5                      conda-forge
2023-12-23T02:12:08.0780939Z     c-ares:                   1.24.0-hd590300_0                     conda-forge
2023-12-23T02:12:08.0781317Z     ca-certificates:          2023.11.17-hbcca054_0                 conda-forge
2023-12-23T02:12:08.0781695Z     cached-property:          1.5.2-hd8ed1ab_1                      conda-forge
2023-12-23T02:12:08.0782062Z     cached_property:          1.5.2-pyha770c72_1                    conda-forge
2023-12-23T02:12:08.0782613Z     cachetools:               5.3.2-pyhd8ed1ab_0                    conda-forge
2023-12-23T02:12:08.0783040Z     certifi:                  2023.11.17-pyhd8ed1ab_0               conda-forge
2023-12-23T02:12:08.0783402Z     cffi:                     1.16.0-py310h2fee648_0                conda-forge
2023-12-23T02:12:08.0783776Z     charset-normalizer:       3.3.2-pyhd8ed1ab_0                    conda-forge
2023-12-23T02:12:08.0784155Z     click:                    8.1.7-unix_pyh707e725_0               conda-forge
2023-12-23T02:12:08.0784521Z     cmake:                    3.28.1-hcfe8598_0                     conda-forge
2023-12-23T02:12:08.0784892Z     contourpy:                1.2.0-py310hd41b1e2_0                 conda-forge
2023-12-23T02:12:08.0785269Z     cryptography:             41.0.7-py310hb8475ec_1                conda-forge
2023-12-23T02:12:08.0785630Z     cuda-version:             11.8-h70ddcb2_2                       conda-forge
2023-12-23T02:12:08.0786001Z     cudatoolkit:              11.8.0-h4ba93d1_12                    conda-forge
2023-12-23T02:12:08.0786372Z     cudnn:                    8.8.0.121-hcdd5f01_4                  conda-forge
2023-12-23T02:12:08.0786724Z     cycler:                   0.12.1-pyhd8ed1ab_0                   conda-forge
2023-12-23T02:12:08.0787106Z     dargs:                    0.4.3-pyhd8ed1ab_0                    conda-forge
2023-12-23T02:12:08.0787465Z     deepmd-kit:               2.2.7-cuda112py310haa36818_4          local      
2023-12-23T02:12:08.0787850Z     f90wrap:                  0.2.13-py310h03ae0af_2                conda-forge
2023-12-23T02:12:08.0788351Z     fftw:                     3.3.10-nompi_hc118613_108             conda-forge
2023-12-23T02:12:08.0788704Z     flask:                    3.0.0-pyhd8ed1ab_0                    conda-forge
2023-12-23T02:12:08.0789072Z     flatbuffers:              23.5.26-h59595ed_1                    conda-forge
2023-12-23T02:12:08.0789443Z     fonttools:                4.47.0-py310h2372a71_0                conda-forge
2023-12-23T02:12:08.0789797Z     freetype:                 2.12.1-h267a509_2                     conda-forge
2023-12-23T02:12:08.0790172Z     frozenlist:               1.4.1-py310h2372a71_0                 conda-forge
2023-12-23T02:12:08.0790542Z     gast:                     0.5.4-pyhd8ed1ab_0                    conda-forge
2023-12-23T02:12:08.0790899Z     gcc_impl_linux-64:        11.4.0-h7aa1c59_3                     conda-forge
2023-12-23T02:12:08.0791274Z     gcc_linux-64:             11.4.0-hfd045f2_2                     conda-forge
2023-12-23T02:12:08.0791734Z     gettext:                  0.21.1-h27087fc_0                     conda-forge
2023-12-23T02:12:08.0792096Z     gfortran_impl_linux-64:   11.4.0-h86428dc_3                     conda-forge
2023-12-23T02:12:08.0792477Z     gfortran_linux-64:        11.4.0-h3047280_2                     conda-forge
2023-12-23T02:12:08.0792834Z     giflib:                   5.2.1-h0b41bf4_3                      conda-forge
2023-12-23T02:12:08.0793140Z     google-auth:              2.25.2-pyhca7485f_0                   conda-forge
2023-12-23T02:12:08.0793512Z     google-auth-oauthlib:     1.2.0-pyhd8ed1ab_0                    conda-forge
2023-12-23T02:12:08.0793878Z     google-pasta:             0.2.0-pyh8c360ce_0                    conda-forge
2023-12-23T02:12:08.0794253Z     grpcio:                   1.59.3-py310h1b8f574_0                conda-forge
2023-12-23T02:12:08.0794617Z     gsl:                      2.7-he838d99_0                        conda-forge
2023-12-23T02:12:08.0794979Z     gxx_impl_linux-64:        11.4.0-h7aa1c59_3                     conda-forge
2023-12-23T02:12:08.0795353Z     gxx_linux-64:             11.4.0-hfc1ae95_2                     conda-forge
2023-12-23T02:12:08.0795737Z     h5py:                     3.10.0-nompi_py310h65828d5_101        conda-forge
2023-12-23T02:12:08.0796097Z     hdf4:                     4.2.15-h2a13503_7                     conda-forge
2023-12-23T02:12:08.0796467Z     hdf5:                     1.14.3-nompi_h4f84152_100             conda-forge
2023-12-23T02:12:08.0796832Z     icu:                      73.2-h59595ed_0                       conda-forge
2023-12-23T02:12:08.0797176Z     idna:                     3.6-pyhd8ed1ab_0                      conda-forge
2023-12-23T02:12:08.0797546Z     importlib-metadata:       7.0.1-pyha770c72_0                    conda-forge
2023-12-23T02:12:08.0797912Z     importlib_metadata:       7.0.1-hd8ed1ab_0                      conda-forge
2023-12-23T02:12:08.0798287Z     itsdangerous:             2.1.2-pyhd8ed1ab_0                    conda-forge
2023-12-23T02:12:08.0798663Z     jinja2:                   3.1.2-pyhd8ed1ab_1                    conda-forge
2023-12-23T02:12:08.0799016Z     keras:                    2.15.0-pyhd8ed1ab_0                   conda-forge
2023-12-23T02:12:08.0799392Z     kernel-headers_linux-64:  2.6.32-he073ed8_16                    conda-forge
2023-12-23T02:12:08.0799768Z     keyutils:                 1.6.1-h166bdaf_0                      conda-forge
2023-12-23T02:12:08.0800134Z     kim-api:                  2.3.0-h097297d_1                      conda-forge
2023-12-23T02:12:08.0800510Z     kiwisolver:               1.4.5-py310hd41b1e2_1                 conda-forge
2023-12-23T02:12:08.0800879Z     krb5:                     1.21.2-h659d440_0                     conda-forge
2023-12-23T02:12:08.0801244Z     lammps:                   2023.08.02-cpu_py310_h2521045_mpich_7 conda-forge
2023-12-23T02:12:08.0801622Z     lcms2:                    2.16-hb7c19ff_0                       conda-forge
2023-12-23T02:12:08.0802066Z     ld_impl_linux-64:         2.40-h41732ed_0                       conda-forge
2023-12-23T02:12:08.0802421Z     lerc:                     4.0.0-h27087fc_0                      conda-forge
2023-12-23T02:12:08.0802792Z     libabseil:                20230802.1-cxx17_h59595ed_0           conda-forge
2023-12-23T02:12:08.0803150Z     libaec:                   1.1.2-h59595ed_1                      conda-forge
2023-12-23T02:12:08.0803519Z     libblas:                  3.9.0-20_linux64_openblas             conda-forge
2023-12-23T02:12:08.0803894Z     libbrotlicommon:          1.1.0-hd590300_1                      conda-forge
2023-12-23T02:12:08.0804253Z     libbrotlidec:             1.1.0-hd590300_1                      conda-forge
2023-12-23T02:12:08.0804624Z     libbrotlienc:             1.1.0-hd590300_1                      conda-forge
2023-12-23T02:12:08.0804998Z     libcblas:                 3.9.0-20_linux64_openblas             conda-forge
2023-12-23T02:12:08.0805433Z     libcurl:                  8.5.0-hca28451_0                      conda-forge
2023-12-23T02:12:08.0805807Z     libdeflate:               1.19-hd590300_0                       conda-forge
2023-12-23T02:12:08.0806184Z     libedit:                  3.1.20191231-he28a2e2_2               conda-forge
2023-12-23T02:12:08.0806537Z     libev:                    4.33-hd590300_2                       conda-forge
2023-12-23T02:12:08.0806901Z     libexpat:                 2.5.0-hcb278e6_1                      conda-forge
2023-12-23T02:12:08.0807267Z     libffi:                   3.4.2-h7f98852_5                      conda-forge
2023-12-23T02:12:08.0807632Z     libgcc-devel_linux-64:    11.4.0-h922705a_103                   conda-forge
2023-12-23T02:12:08.0808011Z     libgcc-ng:                13.2.0-h807b86a_3                     conda-forge
2023-12-23T02:12:08.0808381Z     libgfortran-ng:           13.2.0-h69a702a_3                     conda-forge
2023-12-23T02:12:08.0808740Z     libgfortran5:             13.2.0-ha4646dd_3                     conda-forge
2023-12-23T02:12:08.0809114Z     libgomp:                  13.2.0-h807b86a_3                     conda-forge
2023-12-23T02:12:08.0809467Z     libgrpc:                  1.59.3-hd6c4280_0                     conda-forge
2023-12-23T02:12:08.0809832Z     libiconv:                 1.17-hd590300_2                       conda-forge
2023-12-23T02:12:08.0810196Z     libidn2:                  2.3.4-h166bdaf_0                      conda-forge
2023-12-23T02:12:08.0810548Z     libjpeg-turbo:            3.0.0-hd590300_1                      conda-forge
2023-12-23T02:12:08.0810921Z     liblapack:                3.9.0-20_linux64_openblas             conda-forge
2023-12-23T02:12:08.0811299Z     libnetcdf:                4.9.2-nompi_h9612171_113              conda-forge
2023-12-23T02:12:08.0811662Z     libnghttp2:               1.58.0-h47da74e_1                     conda-forge
2023-12-23T02:12:08.0812035Z     libnsl:                   2.0.1-hd590300_0                      conda-forge
2023-12-23T02:12:08.0812418Z     libopenblas:              0.3.25-pthreads_h413a1c8_0            conda-forge
2023-12-23T02:12:08.0812779Z     libpng:                   1.6.39-h753d276_0                     conda-forge
2023-12-23T02:12:08.0813163Z     libprotobuf:              4.24.4-hf27288f_0                     conda-forge
2023-12-23T02:12:08.0813538Z     libre2-11:                2023.06.02-h7a70373_0                 conda-forge
2023-12-23T02:12:08.0813898Z     libsanitizer:             11.4.0-h4dcbe23_3                     conda-forge
2023-12-23T02:12:08.0814268Z     libsqlite:                3.44.2-h2797004_0                     conda-forge
2023-12-23T02:12:08.0814624Z     libssh2:                  1.11.0-h0841786_0                     conda-forge
2023-12-23T02:12:08.0815004Z     libstdcxx-devel_linux-64: 11.4.0-h922705a_103                   conda-forge
2023-12-23T02:12:08.0815381Z     libstdcxx-ng:             13.2.0-h7e041cc_3                     conda-forge
2023-12-23T02:12:08.0815735Z     libtiff:                  4.6.0-ha9c0a0a_2                      conda-forge
2023-12-23T02:12:08.0816180Z     libunistring:             0.9.10-h7f98852_0                     conda-forge
2023-12-23T02:12:08.0816548Z     libuuid:                  2.38.1-h0b41bf4_0                     conda-forge
2023-12-23T02:12:08.0816899Z     libuv:                    1.46.0-hd590300_0                     conda-forge
2023-12-23T02:12:08.0817265Z     libwebp-base:             1.3.2-hd590300_0                      conda-forge
2023-12-23T02:12:08.0817632Z     libxcb:                   1.15-h0b41bf4_0                       conda-forge
2023-12-23T02:12:08.0817982Z     libxml2:                  2.12.3-h232c23b_0                     conda-forge
2023-12-23T02:12:08.0818351Z     libzip:                   1.10.1-h2629f0a_3                     conda-forge
2023-12-23T02:12:08.0818718Z     libzlib:                  1.2.13-hd590300_5                     conda-forge
2023-12-23T02:12:08.0819066Z     lz4-c:                    1.9.4-hcb278e6_0                      conda-forge
2023-12-23T02:12:08.0819507Z     make:                     4.3-hd18ef5c_1                        conda-forge
2023-12-23T02:12:08.0819876Z     markdown:                 3.5.1-pyhd8ed1ab_0                    conda-forge
2023-12-23T02:12:08.0820238Z     markupsafe:               2.1.3-py310h2372a71_1                 conda-forge
2023-12-23T02:12:08.0820617Z     matplotlib-base:          3.8.2-py310h62c0568_0                 conda-forge
2023-12-23T02:12:08.0820980Z     ml_dtypes:                0.2.0-py310hcc13569_2                 conda-forge
2023-12-23T02:12:08.0821351Z     mlip:                     2.0-hd0d41c4_0                        conda-forge
2023-12-23T02:12:08.0821702Z     mpi:                      1.0-mpich                             conda-forge
2023-12-23T02:12:08.0822048Z     mpich:                    4.1.2-h846660c_101                    conda-forge
2023-12-23T02:12:08.0822421Z     multidict:                6.0.4-py310h2372a71_1                 conda-forge
2023-12-23T02:12:08.0824467Z     munkres:                  1.1.4-pyh9f0ad1d_0                    conda-forge
2023-12-23T02:12:08.0824939Z     n2p2:                     2.2.0-py310hae4c411_4                 conda-forge
2023-12-23T02:12:08.0825408Z     nccl:                     2.19.4.1-h6103f9b_0                   conda-forge
2023-12-23T02:12:08.0825874Z     ncurses:                  6.4-h59595ed_2                        conda-forge
2023-12-23T02:12:08.0826324Z     numpy:                    1.26.2-py310hb13e2d6_0                conda-forge
2023-12-23T02:12:08.0826792Z     oauthlib:                 3.2.2-pyhd8ed1ab_0                    conda-forge
2023-12-23T02:12:08.0827260Z     openjpeg:                 2.5.0-h488ebb8_3                      conda-forge
2023-12-23T02:12:08.0827710Z     openssl:                  3.2.0-hd590300_1                      conda-forge
2023-12-23T02:12:08.0828174Z     opt_einsum:               3.3.0-pyhc1e730c_2                    conda-forge
2023-12-23T02:12:08.0828629Z     packaging:                23.2-pyhd8ed1ab_0                     conda-forge
2023-12-23T02:12:08.0829104Z     pillow:                   10.1.0-py310h01dd4db_0                conda-forge
2023-12-23T02:12:08.0829573Z     pip:                      23.3.2-pyhd8ed1ab_0                   conda-forge
2023-12-23T02:12:08.0830024Z     pkg-config:               0.29.2-h36c2ea0_1008                  conda-forge
2023-12-23T02:12:08.0830512Z     protobuf:                 4.24.4-py310h620c231_0                conda-forge
2023-12-23T02:12:08.0830991Z     pthread-stubs:            0.4-h36c2ea0_1001                     conda-forge
2023-12-23T02:12:08.0831445Z     pyasn1:                   0.5.1-pyhd8ed1ab_0                    conda-forge
2023-12-23T02:12:08.0831915Z     pyasn1-modules:           0.3.0-pyhd8ed1ab_0                    conda-forge
2023-12-23T02:12:08.0832384Z     pycparser:                2.21-pyhd8ed1ab_0                     conda-forge
2023-12-23T02:12:08.0832836Z     pyjwt:                    2.8.0-pyhd8ed1ab_0                    conda-forge
2023-12-23T02:12:08.0833449Z     pyopenssl:                23.3.0-pyhd8ed1ab_0                   conda-forge
2023-12-23T02:12:08.0833938Z     pyparsing:                3.1.1-pyhd8ed1ab_0                    conda-forge
2023-12-23T02:12:08.0834393Z     pysocks:                  1.7.1-pyha2e5f31_6                    conda-forge
2023-12-23T02:12:08.0834864Z     python:                   3.10.13-hd12c33a_0_cpython            conda-forge
2023-12-23T02:12:08.0835323Z     python-dateutil:          2.8.2-pyhd8ed1ab_0                    conda-forge
2023-12-23T02:12:08.0835805Z     python-flatbuffers:       23.5.26-pyhd8ed1ab_0                  conda-forge
2023-12-23T02:12:08.0836282Z     python-hostlist:          1.21-pyh9f0ad1d_0                     conda-forge
2023-12-23T02:12:08.0836742Z     python_abi:               3.10-4_cp310                          conda-forge
2023-12-23T02:12:08.0837207Z     pyu2f:                    0.1.5-pyhd8ed1ab_0                    conda-forge
2023-12-23T02:12:08.0837677Z     pyyaml:                   6.0.1-py310h2372a71_1                 conda-forge
2023-12-23T02:12:08.0838247Z     quippy:                   0.9.12-py310h1dff17b_1                conda-forge
2023-12-23T02:12:08.0838727Z     re2:                      2023.06.02-h2873b5e_0                 conda-forge
2023-12-23T02:12:08.0839192Z     readline:                 8.2-h8228510_1                        conda-forge
2023-12-23T02:12:08.0839644Z     requests:                 2.31.0-pyhd8ed1ab_0                   conda-forge
2023-12-23T02:12:08.0840113Z     requests-oauthlib:        1.3.1-pyhd8ed1ab_0                    conda-forge
2023-12-23T02:12:08.0840584Z     rhash:                    1.4.4-hd590300_0                      conda-forge
2023-12-23T02:12:08.0841037Z     rsa:                      4.9-pyhd8ed1ab_0                      conda-forge
2023-12-23T02:12:08.0841507Z     scipy:                    1.11.4-py310hb13e2d6_0                conda-forge
2023-12-23T02:12:08.0841976Z     setuptools:               68.2.2-pyhd8ed1ab_0                   conda-forge
2023-12-23T02:12:08.0842433Z     six:                      1.16.0-pyh6c4a22f_0                   conda-forge
2023-12-23T02:12:08.0842895Z     snappy:                   1.1.10-h9fff704_0                     conda-forge
2023-12-23T02:12:08.0843374Z     sysroot_linux-64:         2.12-he073ed8_16                      conda-forge
2023-12-23T02:12:08.0843851Z     tensorboard:              2.15.1-pyhd8ed1ab_0                   conda-forge
2023-12-23T02:12:08.0844336Z     tensorboard-data-server:  0.7.0-py310h75e40e8_1                 conda-forge
2023-12-23T02:12:08.0844808Z     tensorflow:               2.15.0-cuda118py310h943c599_1         conda-forge
2023-12-23T02:12:08.0845294Z     tensorflow-base:          2.15.0-cuda118py310h2943ed3_1         conda-forge
2023-12-23T02:12:08.0845786Z     tensorflow-estimator:     2.15.0-cuda118py310ha99e61b_1         conda-forge
2023-12-23T02:12:08.0846252Z     termcolor:                2.3.0-pyhd8ed1ab_0                    conda-forge
2023-12-23T02:12:08.0846724Z     tk:                       8.6.13-noxft_h4845f30_101             conda-forge
2023-12-23T02:12:08.0847198Z     typeguard:                4.1.5-pyhd8ed1ab_1                    conda-forge
2023-12-23T02:12:08.0847650Z     typing-extensions:        4.9.0-hd8ed1ab_0                      conda-forge
2023-12-23T02:12:08.0848126Z     typing_extensions:        4.9.0-pyha770c72_0                    conda-forge
2023-12-23T02:12:08.0848708Z     tzdata:                   2023d-h0c530f3_0                      conda-forge
2023-12-23T02:12:08.0849164Z     unicodedata2:             15.1.0-py310h2372a71_0                conda-forge
2023-12-23T02:12:08.0849633Z     urllib3:                  2.1.0-pyhd8ed1ab_0                    conda-forge
2023-12-23T02:12:08.0850083Z     voro:                     0.4.6-h00ab1b0_0                      conda-forge
2023-12-23T02:12:08.0850541Z     wcmatch:                  8.5-pyhd8ed1ab_0                      conda-forge
2023-12-23T02:12:08.0851008Z     werkzeug:                 3.0.1-pyhd8ed1ab_0                    conda-forge
2023-12-23T02:12:08.0851606Z     wget:                     1.20.3-ha35d2d1_1                     conda-forge
2023-12-23T02:12:08.0852001Z     wheel:                    0.42.0-pyhd8ed1ab_0                   conda-forge
2023-12-23T02:12:08.0863156Z     wrapt:                    1.14.1-py310h5764c6d_1                conda-forge
2023-12-23T02:12:08.7900587Z     xorg-libxau:              1.0.11-hd590300_0                     conda-forge
2023-12-23T02:12:08.7903353Z     xorg-libxdmcp:            1.1.3-h7f98852_0                      conda-forge
2023-12-23T02:12:08.7903980Z     xz:                       5.2.6-h166bdaf_0                      conda-forge
2023-12-23T02:12:08.7904411Z     yaml:                     0.2.5-h7f98852_2                      conda-forge
2023-12-23T02:12:08.7904856Z     yarl:                     1.9.3-py310h2372a71_0                 conda-forge
2023-12-23T02:12:08.7905295Z     zipp:                     3.17.0-pyhd8ed1ab_0                   conda-forge
2023-12-23T02:12:08.7905981Z     zlib:                     1.2.13-hd590300_5                     conda-forge
2023-12-23T02:12:08.7906420Z     zstd:                     1.5.5-hfc55251_0                      conda-forge

Environment info

2023-12-23T01:55:21.6920051Z      active environment : base
2023-12-23T01:55:21.6920500Z     active env location : /opt/conda
2023-12-23T01:55:21.6920729Z             shell level : 1
2023-12-23T01:55:21.6920930Z        user config file : /home/conda/.condarc
2023-12-23T01:55:21.6921160Z  populated config files : /opt/conda/.condarc
2023-12-23T01:55:21.6921361Z                           /home/conda/.condarc
2023-12-23T01:55:21.6921569Z           conda version : 23.11.0
2023-12-23T01:55:21.6921979Z     conda-build version : 3.28.2
2023-12-23T01:55:21.6922189Z          python version : 3.10.13.final.0
2023-12-23T01:55:21.6922409Z                  solver : libmamba (default)
2023-12-23T01:55:21.6922618Z        virtual packages : __archspec=1=x86_64_v4
2023-12-23T01:55:21.6922837Z                           __conda=23.11.0=0
2023-12-23T01:55:21.6923024Z                           __cuda=11.2=0
2023-12-23T01:55:21.6923222Z                           __glibc=2.17=0
2023-12-23T01:55:21.6923408Z                           __linux=6.2.0=0
2023-12-23T01:55:21.6923603Z                           __unix=0=0
2023-12-23T01:55:21.6923808Z        base environment : /opt/conda  (writable)
2023-12-23T01:55:21.6924037Z       conda av data dir : /opt/conda/etc/conda
2023-12-23T01:55:21.6924254Z   conda av metadata url : None
2023-12-23T01:55:21.6924574Z            channel URLs : https://conda.anaconda.org/conda-forge/linux-64
2023-12-23T01:55:21.6924911Z                           https://conda.anaconda.org/conda-forge/noarch
2023-12-23T01:55:21.6925155Z           package cache : /home/conda/feedstock_root/build_artifacts/pkg_cache
2023-12-23T01:55:21.6925386Z                           /opt/conda/pkgs
2023-12-23T01:55:21.6925579Z        envs directories : /opt/conda/envs
2023-12-23T01:55:21.6925789Z                           /home/conda/.conda/envs
2023-12-23T01:55:21.6925978Z                platform : linux-64
2023-12-23T01:55:21.6926471Z              user-agent : conda/23.11.0 requests/2.31.0 CPython/3.10.13 Linux/6.2.0-1018-azure centos/7.9.2009 glibc/2.17 solver/libmamba conda-libmamba-solver/23.12.0 libmambapy/1.5.6
2023-12-23T01:55:21.6926814Z                 UID:GID : 1001:1001
2023-12-23T01:55:21.6927008Z              netrc file : None
2023-12-23T01:55:21.6927207Z            offline mode : False
hmaarrfk commented 6 months ago

can you try the old protobuf, just pin it in the recipe to see

njzjz commented 6 months ago

can you try the old protobuf, just pin it in the recipe to see

libprotobuf 3.21 has the same issue. So I am not sure what the problem is

njzjz commented 6 months ago

https://github.com/tensorflow/tensorflow/pull/51450 says a similar error happens when tensorflow and protobuf use the same instance of libprotobuf. It was fixed, but it seems the issue reappears...

njzjz commented 6 months ago

I confirm it is an upstream issue. I submit https://github.com/tensorflow/tensorflow/pull/62684 to fix it. Not sure what we should do for conda-forge packages.

hmaarrfk commented 6 months ago

We could rebuild.....

njzjz commented 6 months ago

I patch the downstream package to load _pywrap_tensorflow_internal.so before importing tensorflow, which could bypass this error.

# workaround to fix https://github.com/conda-forge/tensorflow-feedstock/issues/364
import ctypes
import importlib
from pathlib import (
    Path,
)

module_spec = importlib.util.find_spec("tensorflow")
ctypes.CDLL(
    str(Path(module_spec.origin).parent / "python" / "_pywrap_tensorflow_internal.so")
)
hmaarrfk commented 6 months ago

Should we rebuild and include this fix. I forgot about this error

xhochy commented 6 months ago

Is there anything we could add to the recipe tests to ensure this is fixed?

njzjz commented 6 months ago

@xhochy The code below should reproduce the error:

import tensorflow as tf

graph = tf.function(lambda x:x).get_concrete_function(1.).graph
tf.compat.v1.train.export_meta_graph(graph=graph, graph_def=graph.as_graph_def())

In a single line:

python -c "import tensorflow as tf;graph = tf.function(lambda x:x).get_concrete_function(1.).graph;tf.compat.v1.train.export_meta_graph(graph=graph,graph_def=graph.as_graph_def())"