rpautrat / SuperPoint

Efficient neural feature detector and descriptor
MIT License
1.95k stars 424 forks source link

About Step1 Training MagicPoint on Synthetic Shapes Problem #282

Open LSK0821 opened 1 year ago

LSK0821 commented 1 year ago

Dear Author, Hello! I am using TF1.2 and python3.6 under linux system.Having an out-of-index problem while executing step 1 to extract gaussian_noise. I changed line 184 in the file /superpoint/superpoint/synthetic_shapes.py to tf.contrib.data.Dataset.map_parallel = lambda self, fn: self.map( fn, num_parallel_calls=config['num_parallel_calls']), because I see that tf1.4 and below use data with contrib, the other parts have not been modified, but look at the content of the error report from this sentence into. The following is the content of the console error reporting:

(test) liusikang@4029GP-TRT:~/cord/superpoint/superpoint$ python experiment.py train configs/magic-point_shapes.yaml magic-point_synth /home/liusikang/anaconda3/envs/test/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:458: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'. _np_qint8 = np.dtype([("qint8", np.int8, 1)]) /home/liusikang/anaconda3/envs/test/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:459: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'. _np_quint8 = np.dtype([("quint8", np.uint8, 1)]) /home/liusikang/anaconda3/envs/test/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:460: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'. _np_qint16 = np.dtype([("qint16", np.int16, 1)]) /home/liusikang/anaconda3/envs/test/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:461: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'. _np_quint16 = np.dtype([("quint16", np.uint16, 1)]) /home/liusikang/anaconda3/envs/test/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:462: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'. _np_qint32 = np.dtype([("qint32", np.int32, 1)]) /home/liusikang/anaconda3/envs/test/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:465: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'. np_resource = np.dtype([("resource", np.ubyte, 1)]) experiment.py:155: YAMLLoadWarning: calling yaml.load() without Loader=... is deprecated, as the default Loader is unsafe. Please read https://msg.pyyaml.org/load for full details. config = yaml.load(f) [12/05/2022 10:17:55 INFO] Running command TRAIN [12/05/2022 10:17:55 INFO] Number of GPUs detected: 1 INFO:tensorflow:Extracting archive for primitive draw_lines. [12/05/2022 10:17:58 INFO] Extracting archive for primitive draw_lines. INFO:tensorflow:Extracting archive for primitive draw_polygon. [12/05/2022 10:18:02 INFO] Extracting archive for primitive draw_polygon. INFO:tensorflow:Extracting archive for primitive draw_multiple_polygons. [12/05/2022 10:18:05 INFO] Extracting archive for primitive draw_multiple_polygons. INFO:tensorflow:Extracting archive for primitive draw_ellipses. [12/05/2022 10:18:08 INFO] Extracting archive for primitive draw_ellipses. INFO:tensorflow:Extracting archive for primitive draw_star. [12/05/2022 10:18:11 INFO] Extracting archive for primitive draw_star. INFO:tensorflow:Extracting archive for primitive draw_checkerboard. [12/05/2022 10:18:14 INFO] Extracting archive for primitive draw_checkerboard. INFO:tensorflow:Extracting archive for primitive draw_stripes. [12/05/2022 10:18:17 INFO] Extracting archive for primitive draw_stripes. INFO:tensorflow:Extracting archive for primitive draw_cube. [12/05/2022 10:18:20 INFO] Extracting archive for primitive draw_cube. INFO:tensorflow:Extracting archive for primitive gaussian_noise. [12/05/2022 10:18:23 INFO] Extracting archive for primitive gaussian_noise. Traceback (most recent call last): File "experiment.py", line 162, in args.func(config, output_dir, args) File "experiment.py", line 99, in _cli_train train(config, config['train_iter'], output_dir, pretrained_dir) File "experiment.py", line 22, in train with _init_graph(config) as net: File "/home/liusikang/anaconda3/envs/test/lib/python3.6/contextlib.py", line 82, in enter return next(self.gen) File "experiment.py", line 73, in _init_graph dataset = get_dataset(config['data']['name'])(config['data']) File "/home/liusikang/cord/superpoint/superpoint/datasets/base_dataset.py", line 108, in init self.tf_splits[n] = self._get_data(self.dataset, n, self.config) File "/home/liusikang/cord/superpoint/superpoint/datasets/synthetic_shapes.py", line 191, in _get_data (filenames[split_name]['images'], filenames[split_name]['points'])).batch(batch_size=32) File "/home/liusikang/anaconda3/envs/test/lib/python3.6/site-packages/tensorflow/contrib/data/python/ops/dataset_ops.py", line 473, in from_tensor_slices return TensorSliceDataset(tensors) File "/home/liusikang/anaconda3/envs/test/lib/python3.6/site-packages/tensorflow/contrib/data/python/ops/dataset_ops.py", line 896, in init batch_dim = flat_tensors[0].get_shape()[0] File "/home/liusikang/anaconda3/envs/test/lib/python3.6/site-packages/tensorflow/python/framework/tensor_shape.py", line 500, in getitem return self._dims[key] IndexError: list index out of range

Looking forward to your reply!

LSK0821 commented 1 year ago

My apologies. I tried to specify the batch size by adding .batch(batch_size=32) "/home/liusikang/cord/superpoint/superpoint/datasets/synthetic_shapes.py", line 191, in _get_data (filenames[split_name]['images'], filenames[split_name]['points'])).batch(batch_size=32), still no solve. So I removed batch(batch_size=32) and it still showed IndexError: list index out of range.

rpautrat commented 1 year ago

Hi, why are you using such an old version of Tensorflow? I suggest you to upgrade, e.g. to 1.12 as in https://github.com/rpautrat/SuperPoint/issues/173#issue-730896838

vrahnos3 commented 1 year ago

Hello, thanks for the great job, I followed the steps of #173 I have a problem with loss nan after extracting all syntetic shapes. Any ideas? Here is my error message: /home/panosvrach/anaconda3/envs/superpoint/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:523: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'. _np_qint8 = np.dtype([("qint8", np.int8, 1)]) /home/panosvrach/anaconda3/envs/superpoint/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:524: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'. _np_quint8 = np.dtype([("quint8", np.uint8, 1)]) /home/panosvrach/anaconda3/envs/superpoint/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:525: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'. _np_qint16 = np.dtype([("qint16", np.int16, 1)]) /home/panosvrach/anaconda3/envs/superpoint/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:526: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'. _np_quint16 = np.dtype([("quint16", np.uint16, 1)]) /home/panosvrach/anaconda3/envs/superpoint/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:527: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'. _np_qint32 = np.dtype([("qint32", np.int32, 1)]) /home/panosvrach/anaconda3/envs/superpoint/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:532: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'. np_resource = np.dtype([("resource", np.ubyte, 1)]) [02/17/2023 16:50:21 INFO] Running command TRAIN [02/17/2023 16:50:21 INFO] Number of GPUs detected: 1 [02/17/2023 16:50:23 INFO] Extracting archive for primitive draw_lines. [02/17/2023 16:50:26 INFO] Extracting archive for primitive draw_polygon. [02/17/2023 16:50:29 INFO] Extracting archive for primitive draw_multiple_polygons. [02/17/2023 16:50:32 INFO] Extracting archive for primitive draw_ellipses. [02/17/2023 16:50:35 INFO] Extracting archive for primitive draw_star. [02/17/2023 16:50:38 INFO] Extracting archive for primitive draw_checkerboard. [02/17/2023 16:50:41 INFO] Extracting archive for primitive draw_stripes. [02/17/2023 16:50:43 INFO] Extracting archive for primitive draw_cube. [02/17/2023 16:50:46 INFO] Extracting archive for primitive gaussian_noise. [02/17/2023 16:50:50 INFO] Caching data, fist access will take some time. [02/17/2023 16:50:51 INFO] Caching data, fist access will take some time. [02/17/2023 16:50:51 INFO] Caching data, fist access will take some time. 2023-02-17 16:50:51.923674: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: SSE4.1 SSE4.2 AVX AVX2 AVX512F FMA 2023-02-17 16:50:52.270209: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1432] Found device 0 with properties: name: NVIDIA GeForce RTX 3090 major: 8 minor: 6 memoryClockRate(GHz): 1.785 pciBusID: 0000:3b:00.0 totalMemory: 23.70GiB freeMemory: 23.43GiB 2023-02-17 16:50:52.270250: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1511] Adding visible gpu devices: 0 2023-02-17 16:55:07.976783: I tensorflow/core/common_runtime/gpu/gpu_device.cc:982] Device interconnect StreamExecutor with strength 1 edge matrix: 2023-02-17 16:55:07.976819: I tensorflow/core/common_runtime/gpu/gpu_device.cc:988] 0 2023-02-17 16:55:07.976825: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1001] 0: N 2023-02-17 16:55:07.976957: W tensorflow/core/common_runtime/gpu/gpu_bfc_allocator.cc:42] Overriding allow_growth setting because the TF_FORCE_GPU_ALLOW_GROWTH environment variable is set. Original config value was 0. 2023-02-17 16:55:07.976989: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 22711 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce RTX 3090, pci bus id: 0000:3b:00.0, compute capability: 8.6) [02/17/2023 16:55:08 INFO] Scale of 0 disables regularizer. [02/17/2023 16:55:08 INFO] Scale of 0 disables regularizer. [02/17/2023 16:55:08 INFO] Scale of 0 disables regularizer. [02/17/2023 16:55:08 INFO] Scale of 0 disables regularizer. [02/17/2023 16:55:08 INFO] Scale of 0 disables regularizer. [02/17/2023 16:55:08 INFO] Scale of 0 disables regularizer. [02/17/2023 16:55:08 INFO] Scale of 0 disables regularizer. [02/17/2023 16:55:08 INFO] Scale of 0 disables regularizer. [02/17/2023 16:55:08 INFO] Scale of 0 disables regularizer. [02/17/2023 16:55:08 INFO] Scale of 0 disables regularizer. [02/17/2023 16:55:09 INFO] Scale of 0 disables regularizer. [02/17/2023 16:55:09 INFO] Scale of 0 disables regularizer. [02/17/2023 16:55:09 INFO] Scale of 0 disables regularizer. [02/17/2023 16:55:09 INFO] Scale of 0 disables regularizer. [02/17/2023 16:55:09 INFO] Scale of 0 disables regularizer. [02/17/2023 16:55:09 INFO] Scale of 0 disables regularizer. [02/17/2023 16:55:09 INFO] Scale of 0 disables regularizer. [02/17/2023 16:55:09 INFO] Scale of 0 disables regularizer. [02/17/2023 16:55:09 INFO] Scale of 0 disables regularizer. [02/17/2023 16:55:09 INFO] Scale of 0 disables regularizer. [02/17/2023 16:55:09 INFO] Scale of 0 disables regularizer. [02/17/2023 16:55:09 INFO] Scale of 0 disables regularizer. [02/17/2023 16:55:09 INFO] Scale of 0 disables regularizer. [02/17/2023 16:55:09 INFO] Scale of 0 disables regularizer. [02/17/2023 16:55:09 INFO] Scale of 0 disables regularizer. [02/17/2023 16:55:09 INFO] Scale of 0 disables regularizer. [02/17/2023 16:55:09 INFO] Scale of 0 disables regularizer. [02/17/2023 16:55:09 INFO] Scale of 0 disables regularizer. [02/17/2023 16:55:09 INFO] Scale of 0 disables regularizer. [02/17/2023 16:55:09 INFO] Scale of 0 disables regularizer. 2023-02-17 16:55:09.704916: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1511] Adding visible gpu devices: 0 2023-02-17 16:55:09.704969: I tensorflow/core/common_runtime/gpu/gpu_device.cc:982] Device interconnect StreamExecutor with strength 1 edge matrix: 2023-02-17 16:55:09.704973: I tensorflow/core/common_runtime/gpu/gpu_device.cc:988] 0 2023-02-17 16:55:09.704977: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1001] 0: N 2023-02-17 16:55:09.705063: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 22711 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce RTX 3090, pci bus id: 0000:3b:00.0, compute capability: 8.6) [02/17/2023 16:55:14 INFO] Start training [02/17/2023 17:14:51 INFO] Iter 0: loss 4.1759, precision 0.0005, recall 0.0482 /home/panosvrach/Desktop/SuperPoint/superpoint/models/base_model.py:387: RuntimeWarning: Mean of empty slice metrics = {m: np.nanmean(metrics[m], axis=0) for m in metrics} [02/17/2023 17:15:10 INFO] Iter 1000: loss nan, precision nan, recall 0.0000 [02/17/2023 17:15:30 INFO] Iter 2000: loss nan, precision nan, recall 0.0000 [02/17/2023 17:15:49 INFO] Iter 3000: loss nan, precision nan, recall 0.0000 [02/17/2023 17:16:08 INFO] Iter 4000: loss nan, precision nan, recall 0.0000 [02/17/2023 17:16:27 INFO] Iter 5000: loss nan, precision nan, recall 0.0000 [02/17/2023 17:16:47 INFO] Iter 6000: loss nan, precision nan, recall 0.0000 [02/17/2023 17:17:06 INFO] Iter 7000: loss nan, precision nan, recall 0.0000 [02/17/2023 17:17:25 INFO] Iter 8000: loss nan, precision nan, recall 0.0000 [02/17/2023 17:17:44 INFO] Iter 9000: loss nan, precision nan, recall 0.0000 [02/17/2023 17:18:03 INFO] Iter 10000: loss nan, precision nan, recall 0.0000 [02/17/2023 17:18:23 INFO] Iter 11000: loss nan, precision nan, recall 0.0000 [02/17/2023 17:18:42 INFO] Iter 12000: loss nan, precision nan, recall 0.0000 [02/17/2023 17:19:01 INFO] Iter 13000: loss nan, precision nan, recall 0.0000 [02/17/2023 17:19:20 INFO] Iter 14000: loss nan, precision nan, recall 0.0000 [02/17/2023 17:19:39 INFO] Iter 15000: loss nan, precision nan, recall 0.0000 [02/17/2023 17:19:58 INFO] Iter 16000: loss nan, precision nan, recall 0.0000 [02/17/2023 17:20:17 INFO] Iter 17000: loss nan, precision nan, recall 0.0000 [02/17/2023 17:20:37 INFO] Iter 18000: loss nan, precision nan, recall 0.0000 [02/17/2023 17:20:56 INFO] Iter 19000: loss nan, precision nan, recall 0.0000 [02/17/2023 17:21:15 INFO] Iter 20000: loss nan, precision nan, recall 0.0000 [02/17/2023 17:21:34 INFO] Iter 21000: loss nan, precision nan, recall 0.0000 [02/17/2023 17:21:53 INFO] Iter 22000: loss nan, precision nan, recall 0.0000 [02/17/2023 17:22:13 INFO] Iter 23000: loss nan, precision nan, recall 0.0000 [02/17/2023 17:22:32 INFO] Iter 24000: loss nan, precision nan, recall 0.0000 [02/17/2023 17:22:51 INFO] Iter 25000: loss nan, precision nan, recall 0.0000 [02/17/2023 17:23:10 INFO] Iter 26000: loss nan, precision nan, recall 0.0000 [02/17/2023 17:23:30 INFO] Iter 27000: loss nan, precision nan, recall 0.0000 [02/17/2023 17:23:49 INFO] Iter 28000: loss nan, precision nan, recall 0.0000 [02/17/2023 17:24:08 INFO] Iter 29000: loss nan, precision nan, recall 0.0000 [02/17/2023 17:24:27 INFO] Iter 30000: loss nan, precision nan, recall 0.0000 [02/17/2023 17:24:46 INFO] Iter 31000: loss nan, precision nan, recall 0.0000 [02/17/2023 17:25:05 INFO] Iter 32000: loss nan, precision nan, recall 0.0000 [02/17/2023 17:25:25 INFO] Iter 33000: loss nan, precision nan, recall 0.0000 [02/17/2023 17:25:44 INFO] Iter 34000: loss nan, precision nan, recall 0.0000 [02/17/2023 17:26:03 INFO] Iter 35000: loss nan, precision nan, recall 0.0000 [02/17/2023 17:26:22 INFO] Iter 36000: loss nan, precision nan, recall 0.0000 [02/17/2023 17:26:41 INFO] Iter 37000: loss nan, precision nan, recall 0.0000 [02/17/2023 17:27:01 INFO] Iter 38000: loss nan, precision nan, recall 0.0000 [02/17/2023 17:27:20 INFO] Iter 39000: loss nan, precision nan, recall 0.0000 [02/17/2023 17:27:39 INFO] Iter 40000: loss nan, precision nan, recall 0.0000 [02/17/2023 17:27:58 INFO] Iter 41000: loss nan, precision nan, recall 0.0000 [02/17/2023 17:28:18 INFO] Iter 42000: loss nan, precision nan, recall 0.0000 [02/17/2023 17:28:37 INFO] Iter 43000: loss nan, precision nan, recall 0.0000 [02/17/2023 17:28:56 INFO] Iter 44000: loss nan, precision nan, recall 0.0000 [02/17/2023 17:29:15 INFO] Iter 45000: loss nan, precision nan, recall 0.0000 [02/17/2023 17:29:34 INFO] Iter 46000: loss nan, precision nan, recall 0.0000 [02/17/2023 17:29:54 INFO] Iter 47000: loss nan, precision nan, recall 0.0000 [02/17/2023 17:30:13 INFO] Iter 48000: loss nan, precision nan, recall 0.0000 [02/17/2023 17:30:32 INFO] Iter 49000: loss nan, precision nan, recall 0.0000 [02/17/2023 17:30:47 INFO] Training finished [02/17/2023 17:30:47 INFO] Saving checkpoint for iteration #50000 2023-02-17 17:30:47.452008: W tensorflow/core/kernels/data/cache_dataset_ops.cc:770] The calling iterator did not fully read the dataset being cached. In order to avoid unexpected truncation of the dataset, the partially cached contents of the datasetwill be discarded. This can happen if you have an input pipeline similar to dataset.cache().take(k).repeat(). You should use dataset.take(k).cache().repeat() instead.

rpautrat commented 1 year ago

Hi, this seems to be very related to this issue: https://github.com/rpautrat/SuperPoint/issues/189

In this other case, it was due to an incompatibility of ampere technology (SM_86) and old cuda + tf versions. Could it be the same for you?

vrahnos3 commented 1 year ago

I use cuda 11.4 so probably its an incompatibility of cuda-tf version. I will try the pytorch implementation as mentioned in #189 otherwise I will try to downgrade cuda version. Thanks a lot for your fast reply.

1z2213 commented 1 year ago

Dear author, Hello! Hi, I followed your instructions of https://github.com/rpautrat/SuperPoint/issues/173 and now I am trying to run the 1st step. python experiment.py train configs/magic-point_shapes.yaml magic-point_synth. I have the same problem with loss nan after extracting all syntetic shapes. I desperately need your help. Looking forward to your reply. Here are the instructions and all the information displayed when I run the code. sunlab@sunlab-ThinkStation-P520:~$ source ~/anaconda3/bin/activate (base) sunlab@sunlab-ThinkStation-P520:~$ conda create -n linenv python=3.6.3 Collecting package metadata (current_repodata.json): done Solving environment: failed with repodata from current_repodata.json, will retry with next repodata source. Collecting package metadata (repodata.json): done Solving environment: done

==> WARNING: A newer version of conda exists. <== current version: 4.10.1 latest version: 23.5.0

Please update conda by running

$ conda update -n base -c defaults conda

Package Plan

environment location: /home/sunlab/anaconda3/envs/linenv

added / updated specs:

The following packages will be downloaded:

package                    |            build
---------------------------|-----------------
pip-21.2.2                 |   py36h06a4308_0         1.8 MB  defaults
python-3.6.3               |       h6c0c0dc_5        25.5 MB  defaults
setuptools-58.0.4          |   py36h06a4308_0         788 KB  defaults
------------------------------------------------------------
                                       Total:        28.1 MB

The following NEW packages will be INSTALLED:

_libgcc_mutex pkgs/main/linux-64::_libgcc_mutex-0.1-main _openmp_mutex pkgs/main/linux-64::_openmp_mutex-5.1-1_gnu ca-certificates pkgs/main/linux-64::ca-certificates-2023.05.30-h06a4308_0 certifi pkgs/main/linux-64::certifi-2021.5.30-py36h06a4308_0 libedit pkgs/main/linux-64::libedit-3.1.20221030-h5eee18b_0 libffi pkgs/main/linux-64::libffi-3.2.1-hf484d3e_1007 libgcc-ng pkgs/main/linux-64::libgcc-ng-11.2.0-h1234567_1 libgomp pkgs/main/linux-64::libgomp-11.2.0-h1234567_1 libstdcxx-ng pkgs/main/linux-64::libstdcxx-ng-11.2.0-h1234567_1 ncurses pkgs/main/linux-64::ncurses-6.4-h6a678d5_0 openssl pkgs/main/linux-64::openssl-1.0.2u-h7b6447c_0 pip pkgs/main/linux-64::pip-21.2.2-py36h06a4308_0 python pkgs/main/linux-64::python-3.6.3-h6c0c0dc_5 readline pkgs/main/linux-64::readline-7.0-h7b6447c_5 setuptools pkgs/main/linux-64::setuptools-58.0.4-py36h06a4308_0 sqlite pkgs/main/linux-64::sqlite-3.33.0-h62c20be_0 tk pkgs/main/linux-64::tk-8.6.12-h1ccaba5_0 wheel pkgs/main/noarch::wheel-0.37.1-pyhd3eb1b0_0 xz pkgs/main/linux-64::xz-5.4.2-h5eee18b_0 zlib pkgs/main/linux-64::zlib-1.2.13-h5eee18b_0

Proceed ([y]/n)? y

Downloading and Extracting Packages python-3.6.3 | 25.5 MB | ##################################### | 100% setuptools-58.0.4 | 788 KB | ##################################### | 100% pip-21.2.2 | 1.8 MB | ##################################### | 100% Preparing transaction: done Verifying transaction: done Executing transaction: done #

To activate this environment, use

#

$ conda activate linenv

#

To deactivate an active environment, use

#

$ conda deactivate

(base) sunlab@sunlab-ThinkStation-P520:~$ conda activate linenv (linenv) sunlab@sunlab-ThinkStation-P520:~$ conda install tensorflow-gpu=1.13 Collecting package metadata (current_repodata.json): done Solving environment: failed with initial frozen solve. Retrying with flexible solve. Collecting package metadata (repodata.json): done Solving environment: done

==> WARNING: A newer version of conda exists. <== current version: 4.10.1 latest version: 23.5.0

Please update conda by running

$ conda update -n base -c defaults conda

Package Plan

environment location: /home/sunlab/anaconda3/envs/linenv

added / updated specs:

The following NEW packages will be INSTALLED:

_tflow_select pkgs/main/linux-64::_tflow_select-2.1.0-gpu absl-py pkgs/main/noarch::absl-py-0.15.0-pyhd3eb1b0_0 astor pkgs/main/linux-64::astor-0.8.1-py36h06a4308_0 blas pkgs/main/linux-64::blas-1.0-mkl c-ares pkgs/main/linux-64::c-ares-1.19.0-h5eee18b_0 cudatoolkit pkgs/main/linux-64::cudatoolkit-10.0.130-0 cudnn pkgs/main/linux-64::cudnn-7.6.5-cuda10.0_0 cupti pkgs/main/linux-64::cupti-10.0.130-0 dataclasses pkgs/main/noarch::dataclasses-0.8-pyh4f3eec9_6 gast pkgs/main/noarch::gast-0.5.3-pyhd3eb1b0_0 grpcio pkgs/main/linux-64::grpcio-1.14.1-py36h9ba97e2_0 h5py pkgs/main/linux-64::h5py-2.10.0-py36hd6299e0_1 hdf5 pkgs/main/linux-64::hdf5-1.10.6-hb1b8bf9_0 intel-openmp pkgs/main/linux-64::intel-openmp-2022.1.0-h9e868ea_3769 keras-applications pkgs/main/noarch::keras-applications-1.0.8-py_1 keras-preprocessi~ pkgs/main/noarch::keras-preprocessing-1.1.2-pyhd3eb1b0_0 libgfortran-ng pkgs/main/linux-64::libgfortran-ng-7.5.0-ha8ba4b0_17 libgfortran4 pkgs/main/linux-64::libgfortran4-7.5.0-ha8ba4b0_17 libprotobuf pkgs/main/linux-64::libprotobuf-3.17.2-h4ff587b_1 markdown pkgs/main/linux-64::markdown-3.1.1-py36_0 mkl pkgs/main/linux-64::mkl-2020.2-256 mkl-service pkgs/main/linux-64::mkl-service-2.3.0-py36he8ac12f_0 mkl_fft pkgs/main/linux-64::mkl_fft-1.3.0-py36h54f3939_0 mkl_random pkgs/main/linux-64::mkl_random-1.1.1-py36h0573a6f_0 mock pkgs/main/noarch::mock-4.0.3-pyhd3eb1b0_0 numpy pkgs/main/linux-64::numpy-1.19.2-py36h54aff64_0 numpy-base pkgs/main/linux-64::numpy-base-1.19.2-py36hfa32c7d_0 protobuf pkgs/main/linux-64::protobuf-3.17.2-py36h295c915_0 scipy pkgs/main/linux-64::scipy-1.5.2-py36h0b6359f_0 six pkgs/main/noarch::six-1.16.0-pyhd3eb1b0_1 tensorboard pkgs/main/linux-64::tensorboard-1.13.1-py36hf484d3e_0 tensorflow pkgs/main/linux-64::tensorflow-1.13.1-gpu_py36h3991807_0 tensorflow-base pkgs/main/linux-64::tensorflow-base-1.13.1-gpu_py36h8d69cac_0 tensorflow-estima~ pkgs/main/noarch::tensorflow-estimator-1.13.0-py_0 tensorflow-gpu pkgs/main/linux-64::tensorflow-gpu-1.13.1-h0d30ee6_0 termcolor pkgs/main/linux-64::termcolor-1.1.0-py36h06a4308_1 werkzeug pkgs/main/noarch::werkzeug-2.0.3-pyhd3eb1b0_0

Proceed ([y]/n)? y

Preparing transaction: done Verifying transaction: done Executing transaction: done (linenv) sunlab@sunlab-ThinkStation-P520:~$ cd superpoint/SuperPoint-master (linenv) sunlab@sunlab-ThinkStation-P520:~/superpoint/SuperPoint-master$ make install pip3 install -r requirements.txt Requirement already satisfied: numpy in /home/sunlab/anaconda3/envs/linenv/lib/python3.6/site-packages (from -r requirements.txt (line 1)) (1.19.2) Requirement already satisfied: scipy in /home/sunlab/anaconda3/envs/linenv/lib/python3.6/site-packages (from -r requirements.txt (line 2)) (1.5.2) Collecting opencv-python==3.4.2.16 Using cached opencv_python-3.4.2.16-cp36-cp36m-manylinux1_x86_64.whl (25.0 MB) Collecting opencv-contrib-python==3.4.2.16 Using cached opencv_contrib_python-3.4.2.16-cp36-cp36m-manylinux1_x86_64.whl (30.6 MB) Collecting tqdm Using cached tqdm-4.64.1-py2.py3-none-any.whl (78 kB) Collecting pyyaml Using cached PyYAML-6.0-cp36-cp36m-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl (603 kB) Collecting flake8 Using cached flake8-5.0.4-py2.py3-none-any.whl (61 kB) Collecting jupyter Using cached jupyter-1.0.0-py2.py3-none-any.whl (2.7 kB) Collecting importlib-resources Using cached importlib_resources-5.4.0-py3-none-any.whl (28 kB) Collecting mccabe<0.8.0,>=0.7.0 Using cached mccabe-0.7.0-py2.py3-none-any.whl (7.3 kB) Collecting pycodestyle<2.10.0,>=2.9.0 Using cached pycodestyle-2.9.1-py2.py3-none-any.whl (41 kB) Collecting importlib-metadata<4.3,>=1.1.0 Using cached importlib_metadata-4.2.0-py3-none-any.whl (16 kB) Collecting pyflakes<2.6.0,>=2.5.0 Using cached pyflakes-2.5.0-py2.py3-none-any.whl (66 kB) Collecting ipykernel Using cached ipykernel-5.5.6-py3-none-any.whl (121 kB) Collecting nbconvert Using cached nbconvert-6.0.7-py3-none-any.whl (552 kB) Collecting ipywidgets Using cached ipywidgets-7.7.5-py2.py3-none-any.whl (123 kB) Collecting notebook Using cached notebook-6.4.10-py3-none-any.whl (9.9 MB) Collecting qtconsole Using cached qtconsole-5.2.2-py3-none-any.whl (120 kB) Collecting jupyter-console Using cached jupyter_console-6.4.3-py3-none-any.whl (22 kB) Collecting typing-extensions>=3.6.4 Using cached typing_extensions-4.1.1-py3-none-any.whl (26 kB) Collecting zipp>=0.5 Using cached zipp-3.6.0-py3-none-any.whl (5.3 kB) Collecting jupyter-client Using cached jupyter_client-7.1.2-py3-none-any.whl (130 kB) Collecting traitlets>=4.1.0 Using cached traitlets-4.3.3-py2.py3-none-any.whl (75 kB) Collecting ipython>=5.0.0 Using cached ipython-7.16.3-py3-none-any.whl (783 kB) Collecting tornado>=4.2 Using cached tornado-6.1-cp36-cp36m-manylinux2010_x86_64.whl (427 kB) Collecting ipython-genutils Using cached ipython_genutils-0.2.0-py2.py3-none-any.whl (26 kB) Collecting prompt-toolkit!=3.0.0,!=3.0.1,<3.1.0,>=2.0.0 Using cached prompt_toolkit-3.0.36-py3-none-any.whl (386 kB) Collecting pexpect Using cached pexpect-4.8.0-py2.py3-none-any.whl (59 kB) Collecting backcall Using cached backcall-0.2.0-py2.py3-none-any.whl (11 kB) Requirement already satisfied: setuptools>=18.5 in /home/sunlab/anaconda3/envs/linenv/lib/python3.6/site-packages (from ipython>=5.0.0->ipykernel->jupyter->-r requirements.txt (line 8)) (58.0.4) Collecting pygments Using cached Pygments-2.14.0-py3-none-any.whl (1.1 MB) Collecting decorator Using cached decorator-5.1.1-py3-none-any.whl (9.1 kB) Collecting jedi<=0.17.2,>=0.10 Using cached jedi-0.17.2-py2.py3-none-any.whl (1.4 MB) Collecting pickleshare Using cached pickleshare-0.7.5-py2.py3-none-any.whl (6.9 kB) Collecting parso<0.8.0,>=0.7.0 Using cached parso-0.7.1-py2.py3-none-any.whl (109 kB) Collecting wcwidth Using cached wcwidth-0.2.6-py2.py3-none-any.whl (29 kB) Requirement already satisfied: six in /home/sunlab/anaconda3/envs/linenv/lib/python3.6/site-packages (from traitlets>=4.1.0->ipykernel->jupyter->-r requirements.txt (line 8)) (1.16.0) Collecting jupyterlab-widgets<3,>=1.0.0 Using cached jupyterlab_widgets-1.1.4-py3-none-any.whl (246 kB) Collecting widgetsnbextension~=3.6.4 Using cached widgetsnbextension-3.6.4-py2.py3-none-any.whl (1.6 MB) Collecting prometheus-client Using cached prometheus_client-0.17.0-py3-none-any.whl (60 kB) Collecting Send2Trash>=1.8.0 Using cached Send2Trash-1.8.2-py3-none-any.whl (18 kB) Collecting jinja2 Using cached Jinja2-3.0.3-py3-none-any.whl (133 kB) Collecting argon2-cffi Using cached argon2_cffi-21.3.0-py3-none-any.whl (14 kB) Collecting nbformat Using cached nbformat-5.1.3-py3-none-any.whl (178 kB) Collecting nest-asyncio>=1.5 Using cached nest_asyncio-1.5.6-py3-none-any.whl (5.2 kB) Collecting pyzmq>=17 Using cached pyzmq-25.1.0-cp36-cp36m-manylinux_2_5_x86_64.manylinux1_x86_64.whl (1.1 MB) Collecting terminado>=0.8.3 Using cached terminado-0.12.1-py3-none-any.whl (15 kB) Collecting jupyter-core>=4.6.1 Using cached jupyter_core-4.9.2-py3-none-any.whl (86 kB) Collecting python-dateutil>=2.1 Using cached python_dateutil-2.8.2-py2.py3-none-any.whl (247 kB) Collecting entrypoints Using cached entrypoints-0.4-py3-none-any.whl (5.3 kB) Collecting defusedxml Using cached defusedxml-0.7.1-py2.py3-none-any.whl (25 kB) Collecting pandocfilters>=1.4.1 Using cached pandocfilters-1.5.0-py2.py3-none-any.whl (8.7 kB) Collecting testpath Using cached testpath-0.6.0-py3-none-any.whl (83 kB) Collecting mistune<2,>=0.8.1 Using cached mistune-0.8.4-py2.py3-none-any.whl (16 kB) Collecting bleach Using cached bleach-4.1.0-py2.py3-none-any.whl (157 kB) Collecting nbclient<0.6.0,>=0.5.0 Using cached nbclient-0.5.9-py3-none-any.whl (69 kB) Collecting jupyterlab-pygments Using cached jupyterlab_pygments-0.1.2-py2.py3-none-any.whl (4.6 kB) Collecting MarkupSafe>=2.0 Using cached MarkupSafe-2.0.1-cp36-cp36m-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl (30 kB) Collecting async-generator Using cached async_generator-1.10-py3-none-any.whl (18 kB) Collecting jsonschema!=2.5.0,>=2.4 Using cached jsonschema-3.2.0-py2.py3-none-any.whl (56 kB) Collecting attrs>=17.4.0 Using cached attrs-22.2.0-py3-none-any.whl (60 kB) Collecting pyrsistent>=0.14.0 Using cached pyrsistent-0.18.0-cp36-cp36m-manylinux1_x86_64.whl (117 kB) Collecting ptyprocess Using cached ptyprocess-0.7.0-py2.py3-none-any.whl (13 kB) Requirement already satisfied: dataclasses in /home/sunlab/anaconda3/envs/linenv/lib/python3.6/site-packages (from argon2-cffi->notebook->jupyter->-r requirements.txt (line 8)) (0.8) Collecting argon2-cffi-bindings Using cached argon2_cffi_bindings-21.2.0-cp36-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (86 kB) Collecting cffi>=1.0.1 Using cached cffi-1.15.1-cp36-cp36m-manylinux_2_5_x86_64.manylinux1_x86_64.whl (402 kB) Collecting pycparser Using cached pycparser-2.21-py2.py3-none-any.whl (118 kB) Collecting webencodings Using cached webencodings-0.5.1-py2.py3-none-any.whl (11 kB) Collecting packaging Using cached packaging-21.3-py3-none-any.whl (40 kB) Collecting pyparsing!=3.0.5,>=2.0.2 Using cached pyparsing-3.0.7-py3-none-any.whl (98 kB) Collecting qtpy Using cached QtPy-2.0.1-py3-none-any.whl (65 kB) WARNING: The candidate selected for download or install is a yanked version: 'opencv-python' candidate (version 3.4.2.16 at https://files.pythonhosted.org/packages/fa/7d/5042b668a8ed41d2a80b8c172f5efcd572e3c046c75ae029407e19b7fc68/opencv_python-3.4.2.16-cp36-cp36m-manylinux1_x86_64.whl#sha256=d75f60baced5086300a19c8ba63e75d059e8dce333795ef02084b9be6ec61516 (from https://pypi.org/simple/opencv-python/)) Reason for being yanked: Release deprecated WARNING: The candidate selected for download or install is a yanked version: 'opencv-contrib-python' candidate (version 3.4.2.16 at https://files.pythonhosted.org/packages/08/f1/66330f4042c4fb3b2d77a159db8e8916d9cdecc29bc8c1f56bc7f8a9bec9/opencv_contrib_python-3.4.2.16-cp36-cp36m-manylinux1_x86_64.whl#sha256=8de56394a9a3cf8788559032c2139c622ffdc7e37c32215ec865b4e1cd2ca70d (from https://pypi.org/simple/opencv-contrib-python/)) Reason for being yanked: Release deprecated Installing collected packages: zipp, typing-extensions, ipython-genutils, decorator, traitlets, pyrsistent, importlib-metadata, attrs, wcwidth, tornado, pyzmq, python-dateutil, pyparsing, pycparser, ptyprocess, parso, nest-asyncio, jupyter-core, jsonschema, entrypoints, webencodings, pygments, prompt-toolkit, pickleshare, pexpect, packaging, nbformat, MarkupSafe, jupyter-client, jedi, cffi, backcall, async-generator, testpath, pandocfilters, nbclient, mistune, jupyterlab-pygments, jinja2, ipython, defusedxml, bleach, argon2-cffi-bindings, terminado, Send2Trash, prometheus-client, nbconvert, ipykernel, argon2-cffi, notebook, widgetsnbextension, qtpy, jupyterlab-widgets, qtconsole, pyflakes, pycodestyle, mccabe, jupyter-console, ipywidgets, importlib-resources, tqdm, pyyaml, opencv-python, opencv-contrib-python, jupyter, flake8 Successfully installed MarkupSafe-2.0.1 Send2Trash-1.8.2 argon2-cffi-21.3.0 argon2-cffi-bindings-21.2.0 async-generator-1.10 attrs-22.2.0 backcall-0.2.0 bleach-4.1.0 cffi-1.15.1 decorator-5.1.1 defusedxml-0.7.1 entrypoints-0.4 flake8-5.0.4 importlib-metadata-4.2.0 importlib-resources-5.4.0 ipykernel-5.5.6 ipython-7.16.3 ipython-genutils-0.2.0 ipywidgets-7.7.5 jedi-0.17.2 jinja2-3.0.3 jsonschema-3.2.0 jupyter-1.0.0 jupyter-client-7.1.2 jupyter-console-6.4.3 jupyter-core-4.9.2 jupyterlab-pygments-0.1.2 jupyterlab-widgets-1.1.4 mccabe-0.7.0 mistune-0.8.4 nbclient-0.5.9 nbconvert-6.0.7 nbformat-5.1.3 nest-asyncio-1.5.6 notebook-6.4.10 opencv-contrib-python-3.4.2.16 opencv-python-3.4.2.16 packaging-21.3 pandocfilters-1.5.0 parso-0.7.1 pexpect-4.8.0 pickleshare-0.7.5 prometheus-client-0.17.0 prompt-toolkit-3.0.36 ptyprocess-0.7.0 pycodestyle-2.9.1 pycparser-2.21 pyflakes-2.5.0 pygments-2.14.0 pyparsing-3.0.7 pyrsistent-0.18.0 python-dateutil-2.8.2 pyyaml-6.0 pyzmq-25.1.0 qtconsole-5.2.2 qtpy-2.0.1 terminado-0.12.1 testpath-0.6.0 tornado-6.1 tqdm-4.64.1 traitlets-4.3.3 typing-extensions-4.1.1 wcwidth-0.2.6 webencodings-0.5.1 widgetsnbextension-3.6.4 zipp-3.6.0 pip3 install -e . Obtaining file:///home/sunlab/superpoint/SuperPoint-master Installing collected packages: superpoint Running setup.py develop for superpoint Successfully installed superpoint-0.0 sh setup.sh Path of the directory where datasets are stored and read: /home/sunlab//superpoint/SuperPoint-master/DATA_DIR Path of the directory where experiments data (logs, checkpoints, configs) are written: /home/sunlab//superpoint/SuperPoint-master/EXPER_DIR (linenv) sunlab@sunlab-ThinkStation-P520:~/superpoint/SuperPoint-master$ cd ./superpoint (linenv) sunlab@sunlab-ThinkStation-P520:~/superpoint/SuperPoint-master/superpoint$ export TMPDIR=/tmp/ (linenv) sunlab@sunlab-ThinkStation-P520:~/superpoint/SuperPoint-master/superpoint$ export TF_FORCE_GPU_ALLOW_GROWTH=true (linenv) sunlab@sunlab-ThinkStation-P520:~/superpoint/SuperPoint-master/superpoint$ python experiment.py train configs/magic-point_shapes.yaml magic-point_synth/home/sunlab/anaconda3/envs/linenv/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:526: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'. _np_qint8 = np.dtype([("qint8", np.int8, 1)]) /home/sunlab/anaconda3/envs/linenv/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:527: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'. _np_quint8 = np.dtype([("quint8", np.uint8, 1)]) /home/sunlab/anaconda3/envs/linenv/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:528: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'. _np_qint16 = np.dtype([("qint16", np.int16, 1)]) /home/sunlab/anaconda3/envs/linenv/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:529: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'. _np_quint16 = np.dtype([("quint16", np.uint16, 1)]) /home/sunlab/anaconda3/envs/linenv/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:530: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'. _np_qint32 = np.dtype([("qint32", np.int32, 1)]) /home/sunlab/anaconda3/envs/linenv/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:535: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'. np_resource = np.dtype([("resource", np.ubyte, 1)]) [07/06/2023 23:37:58 INFO] Running command TRAIN Traceback (most recent call last): File "experiment.py", line 160, in args.func(config, output_dir, args) File "experiment.py", line 97, in _cli_train train(config, config['train_iter'], output_dir, pretrained_dir) File "experiment.py", line 22, in train with _init_graph(config) as net: File "/home/sunlab/anaconda3/envs/linenv/lib/python3.6/contextlib.py", line 81, in enter return next(self.gen) File "experiment.py", line 68, in _init_graph n_gpus = get_num_gpus() File "experiment.py", line 62, in get_num_gpus return len(os.environ['CUDA_VISIBLE_DEVICES'].split(',')) File "/home/sunlab/anaconda3/envs/linenv/lib/python3.6/os.py", line 669, in getitem raise KeyError(key) from None KeyError: 'CUDA_VISIBLE_DEVICES' (linenv) sunlab@sunlab-ThinkStation-P520:~/superpoint/SuperPoint-master/superpoint$ export CUDA_VISIBLE_DEVICES=0 (linenv) sunlab@sunlab-ThinkStation-P520:~/superpoint/SuperPoint-master/superpoint$ python experiment.py train configs/magic-point_shapes.yaml magic-point_synth/home/sunlab/anaconda3/envs/linenv/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:526: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'. _np_qint8 = np.dtype([("qint8", np.int8, 1)]) /home/sunlab/anaconda3/envs/linenv/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:527: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'. _np_quint8 = np.dtype([("quint8", np.uint8, 1)]) /home/sunlab/anaconda3/envs/linenv/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:528: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'. _np_qint16 = np.dtype([("qint16", np.int16, 1)]) /home/sunlab/anaconda3/envs/linenv/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:529: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'. _np_quint16 = np.dtype([("quint16", np.uint16, 1)]) /home/sunlab/anaconda3/envs/linenv/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:530: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'. _np_qint32 = np.dtype([("qint32", np.int32, 1)]) /home/sunlab/anaconda3/envs/linenv/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:535: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'. np_resource = np.dtype([("resource", np.ubyte, 1)]) [07/06/2023 23:38:09 INFO] Running command TRAIN [07/06/2023 23:38:09 INFO] Number of GPUs detected: 1 [07/06/2023 23:38:12 INFO] Extracting archive for primitive draw_lines. [07/06/2023 23:38:15 INFO] Extracting archive for primitive draw_polygon. [07/06/2023 23:38:19 INFO] Extracting archive for primitive draw_multiple_polygons. [07/06/2023 23:38:22 INFO] Extracting archive for primitive draw_ellipses. [07/06/2023 23:38:26 INFO] Extracting archive for primitive draw_star. [07/06/2023 23:38:30 INFO] Extracting archive for primitive draw_checkerboard. [07/06/2023 23:38:33 INFO] Extracting archive for primitive draw_stripes. [07/06/2023 23:38:36 INFO] Extracting archive for primitive draw_cube. [07/06/2023 23:38:40 INFO] Extracting archive for primitive gaussian_noise. [07/06/2023 23:38:45 WARNING] From /home/sunlab/superpoint/SuperPoint-master/superpoint/datasets/synthetic_shapes.py:189: py_func (from tensorflow.python.ops.script_ops) is deprecated and will be removed in a future version. Instructions for updating: tf.py_func is deprecated in TF V2. Instead, use tf.py_function, which takes a python function which manipulates tf eager tensors instead of numpy arrays. It's easy to convert a tf eager tensor to an ndarray (just call tensor.numpy()) but having access to eager tensors means tf.py_functions can use accelerators such as GPUs as well as being differentiable using a gradient tape.

[07/06/2023 23:38:45 INFO] Caching data, fist access will take some time. [07/06/2023 23:38:45 WARNING] From /home/sunlab/anaconda3/envs/linenv/lib/python3.6/site-packages/tensorflow/python/ops/control_flow_ops.py:423: colocate_with (from tensorflow.python.framework.ops) is deprecated and will be removed in a future version. Instructions for updating: Colocations handled automatically by placer. [07/06/2023 23:38:45 WARNING] From /home/sunlab/superpoint/SuperPoint-master/superpoint/models/homographies.py:218: to_float (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version. Instructions for updating: Use tf.cast instead. [07/06/2023 23:38:45 WARNING] From /home/sunlab/superpoint/SuperPoint-master/superpoint/models/homographies.py:277: to_int32 (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version. Instructions for updating: Use tf.cast instead. [07/06/2023 23:38:46 INFO] Caching data, fist access will take some time. [07/06/2023 23:38:46 INFO] Caching data, fist access will take some time. 2023-07-06 23:38:46.500189: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: SSE4.1 SSE4.2 AVX AVX2 AVX512F FMA 2023-07-06 23:38:46.688675: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 3600000000 Hz 2023-07-06 23:38:46.701881: I tensorflow/compiler/xla/service/service.cc:150] XLA service 0x557760fccac0 executing computations on platform Host. Devices: 2023-07-06 23:38:46.702032: I tensorflow/compiler/xla/service/service.cc:158] StreamExecutor device (0): , 2023-07-06 23:38:46.823689: I tensorflow/compiler/xla/service/service.cc:150] XLA service 0x557760fce720 executing computations on platform CUDA. Devices: 2023-07-06 23:38:46.823750: I tensorflow/compiler/xla/service/service.cc:158] StreamExecutor device (0): NVIDIA GeForce RTX 3060, Compute Capability 8.6 2023-07-06 23:38:46.823998: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1433] Found device 0 with properties: name: NVIDIA GeForce RTX 3060 major: 8 minor: 6 memoryClockRate(GHz): 1.837 pciBusID: 0000:65:00.0 totalMemory: 11.76GiB freeMemory: 10.53GiB 2023-07-06 23:38:46.824031: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1512] Adding visible gpu devices: 0 2023-07-06 23:38:46.845678: I tensorflow/core/common_runtime/gpu/gpu_device.cc:984] Device interconnect StreamExecutor with strength 1 edge matrix: 2023-07-06 23:38:46.845739: I tensorflow/core/common_runtime/gpu/gpu_device.cc:990] 0 2023-07-06 23:38:46.845756: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1003] 0: N 2023-07-06 23:38:46.845924: W tensorflow/core/common_runtime/gpu/gpu_bfc_allocator.cc:42] Overriding allow_growth setting because the TF_FORCE_GPU_ALLOW_GROWTH environment variable is set. Original config value was 0. 2023-07-06 23:38:46.845989: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 10245 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce RTX 3060, pci bus id: 0000:65:00.0, compute capability: 8.6) [07/06/2023 23:38:47 INFO] Scale of 0 disables regularizer. [07/06/2023 23:38:47 WARNING] From /home/sunlab/superpoint/SuperPoint-master/superpoint/models/backbones/vgg.py:10: conv2d (from tensorflow.python.layers.convolutional) is deprecated and will be removed in a future version. Instructions for updating: Use keras.layers.conv2d instead. [07/06/2023 23:38:47 WARNING] From /home/sunlab/superpoint/SuperPoint-master/superpoint/models/backbones/vgg.py:14: batch_normalization (from tensorflow.python.layers.normalization) is deprecated and will be removed in a future version. Instructions for updating: Use keras.layers.batch_normalization instead. [07/06/2023 23:38:47 INFO] Scale of 0 disables regularizer. [07/06/2023 23:38:47 WARNING] From /home/sunlab/superpoint/SuperPoint-master/superpoint/models/backbones/vgg.py:28: max_pooling2d (from tensorflow.python.layers.pooling) is deprecated and will be removed in a future version. Instructions for updating: Use keras.layers.max_pooling2d instead. [07/06/2023 23:38:47 INFO] Scale of 0 disables regularizer. [07/06/2023 23:38:47 INFO] Scale of 0 disables regularizer. [07/06/2023 23:38:47 INFO] Scale of 0 disables regularizer. [07/06/2023 23:38:47 INFO] Scale of 0 disables regularizer. [07/06/2023 23:38:47 INFO] Scale of 0 disables regularizer. [07/06/2023 23:38:47 INFO] Scale of 0 disables regularizer. [07/06/2023 23:38:47 INFO] Scale of 0 disables regularizer. [07/06/2023 23:38:47 INFO] Scale of 0 disables regularizer. [07/06/2023 23:38:48 INFO] Scale of 0 disables regularizer. [07/06/2023 23:38:48 INFO] Scale of 0 disables regularizer. [07/06/2023 23:38:48 INFO] Scale of 0 disables regularizer. [07/06/2023 23:38:48 INFO] Scale of 0 disables regularizer. [07/06/2023 23:38:48 INFO] Scale of 0 disables regularizer. [07/06/2023 23:38:48 INFO] Scale of 0 disables regularizer. [07/06/2023 23:38:48 INFO] Scale of 0 disables regularizer. [07/06/2023 23:38:48 INFO] Scale of 0 disables regularizer. [07/06/2023 23:38:48 INFO] Scale of 0 disables regularizer. [07/06/2023 23:38:48 INFO] Scale of 0 disables regularizer. [07/06/2023 23:38:48 INFO] Scale of 0 disables regularizer. [07/06/2023 23:38:48 INFO] Scale of 0 disables regularizer. [07/06/2023 23:38:48 INFO] Scale of 0 disables regularizer. [07/06/2023 23:38:48 INFO] Scale of 0 disables regularizer. [07/06/2023 23:38:48 INFO] Scale of 0 disables regularizer. [07/06/2023 23:38:48 INFO] Scale of 0 disables regularizer. [07/06/2023 23:38:48 INFO] Scale of 0 disables regularizer. [07/06/2023 23:38:48 INFO] Scale of 0 disables regularizer. [07/06/2023 23:38:48 INFO] Scale of 0 disables regularizer. [07/06/2023 23:38:48 INFO] Scale of 0 disables regularizer. 2023-07-06 23:38:48.493726: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1512] Adding visible gpu devices: 0 2023-07-06 23:38:48.493769: I tensorflow/core/common_runtime/gpu/gpu_device.cc:984] Device interconnect StreamExecutor with strength 1 edge matrix: 2023-07-06 23:38:48.493774: I tensorflow/core/common_runtime/gpu/gpu_device.cc:990] 0 2023-07-06 23:38:48.493778: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1003] 0: N 2023-07-06 23:38:48.493822: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 10245 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce RTX 3060, pci bus id: 0000:65:00.0, compute capability: 8.6) [07/06/2023 23:38:53 INFO] Start training [07/07/2023 00:01:33 INFO] Iter 0: loss 4.1784, precision 0.0006, recall 0.0583 /home/sunlab/superpoint/SuperPoint-master/superpoint/models/base_model.py:387: RuntimeWarning: Mean of empty slice metrics = {m: np.nanmean(metrics[m], axis=0) for m in metrics} [07/07/2023 00:01:45 INFO] Iter 1000: loss nan, precision nan, recall 0.0000 [07/07/2023 00:01:56 INFO] Iter 2000: loss nan, precision nan, recall 0.0000 [07/07/2023 00:02:08 INFO] Iter 3000: loss nan, precision nan, recall 0.0000 [07/07/2023 00:02:20 INFO] Iter 4000: loss nan, precision nan, recall 0.0000 [07/07/2023 00:02:32 INFO] Iter 5000: loss nan, precision nan, recall 0.0000 [07/07/2023 00:02:44 INFO] Iter 6000: loss nan, precision nan, recall 0.0000 [07/07/2023 00:02:56 INFO] Iter 7000: loss nan, precision nan, recall 0.0000 [07/07/2023 00:03:08 INFO] Iter 8000: loss nan, precision nan, recall 0.0000 [07/07/2023 00:03:20 INFO] Iter 9000: loss nan, precision nan, recall 0.0000 [07/07/2023 00:03:32 INFO] Iter 10000: loss nan, precision nan, recall 0.0000 [07/07/2023 00:03:44 INFO] Iter 11000: loss nan, precision nan, recall 0.0000 [07/07/2023 00:03:56 INFO] Iter 12000: loss nan, precision nan, recall 0.0000 [07/07/2023 00:04:08 INFO] Iter 13000: loss nan, precision nan, recall 0.0000 [07/07/2023 00:04:20 INFO] Iter 14000: loss nan, precision nan, recall 0.0000 [07/07/2023 00:04:32 INFO] Iter 15000: loss nan, precision nan, recall 0.0000 [07/07/2023 00:04:44 INFO] Iter 16000: loss nan, precision nan, recall 0.0000 [07/07/2023 00:04:55 INFO] Iter 17000: loss nan, precision nan, recall 0.0000 [07/07/2023 00:05:07 INFO] Iter 18000: loss nan, precision nan, recall 0.0000 [07/07/2023 00:05:19 INFO] Iter 19000: loss nan, precision nan, recall 0.0000 [07/07/2023 00:05:31 INFO] Iter 20000: loss nan, precision nan, recall 0.0000 [07/07/2023 00:05:42 INFO] Iter 21000: loss nan, precision nan, recall 0.0000 [07/07/2023 00:05:54 INFO] Iter 22000: loss nan, precision nan, recall 0.0000 [07/07/2023 00:06:06 INFO] Iter 23000: loss nan, precision nan, recall 0.0000 [07/07/2023 00:06:17 INFO] Iter 24000: loss nan, precision nan, recall 0.0000 [07/07/2023 00:06:29 INFO] Iter 25000: loss nan, precision nan, recall 0.0000 [07/07/2023 00:06:41 INFO] Iter 26000: loss nan, precision nan, recall 0.0000 [07/07/2023 00:06:52 INFO] Iter 27000: loss nan, precision nan, recall 0.0000 [07/07/2023 00:07:04 INFO] Iter 28000: loss nan, precision nan, recall 0.0000 [07/07/2023 00:07:15 INFO] Iter 29000: loss nan, precision nan, recall 0.0000 [07/07/2023 00:07:27 INFO] Iter 30000: loss nan, precision nan, recall 0.0000 [07/07/2023 00:07:39 INFO] Iter 31000: loss nan, precision nan, recall 0.0000 [07/07/2023 00:07:54 INFO] Iter 32000: loss nan, precision nan, recall 0.0000 [07/07/2023 00:08:10 INFO] Iter 33000: loss nan, precision nan, recall 0.0000 [07/07/2023 00:08:41 INFO] Iter 34000: loss nan, precision nan, recall 0.0000 [07/07/2023 00:09:14 INFO] Iter 35000: loss nan, precision nan, recall 0.0000 [07/07/2023 00:09:47 INFO] Iter 36000: loss nan, precision nan, recall 0.0000 [07/07/2023 00:10:15 INFO] Iter 37000: loss nan, precision nan, recall 0.0000 [07/07/2023 00:10:32 INFO] Iter 38000: loss nan, precision nan, recall 0.0000 [07/07/2023 00:10:50 INFO] Iter 39000: loss nan, precision nan, recall 0.0000 [07/07/2023 00:11:06 INFO] Iter 40000: loss nan, precision nan, recall 0.0000 [07/07/2023 00:11:23 INFO] Iter 41000: loss nan, precision nan, recall 0.0000 [07/07/2023 00:11:39 INFO] Iter 42000: loss nan, precision nan, recall 0.0000 [07/07/2023 00:11:57 INFO] Iter 43000: loss nan, precision nan, recall 0.0000 [07/07/2023 00:12:14 INFO] Iter 44000: loss nan, precision nan, recall 0.0000 [07/07/2023 00:12:31 INFO] Iter 45000: loss nan, precision nan, recall 0.0000 [07/07/2023 00:12:49 INFO] Iter 46000: loss nan, precision nan, recall 0.0000 [07/07/2023 00:13:06 INFO] Iter 47000: loss nan, precision nan, recall 0.0000 [07/07/2023 00:13:23 INFO] Iter 48000: loss nan, precision nan, recall 0.0000 [07/07/2023 00:13:41 INFO] Iter 49000: loss nan, precision nan, recall 0.0000 [07/07/2023 00:13:54 INFO] Training finished [07/07/2023 00:13:55 INFO] Saving checkpoint for iteration #50000 2023-07-07 00:13:56.578198: W tensorflow/core/kernels/data/cache_dataset_ops.cc:810] The calling iterator did not fully read the dataset being cached. In order to avoid unexpected truncation of the dataset, the partially cached contents of the datasetwill be discarded. This can happen if you have an input pipeline similar to dataset.cache().take(k).repeat(). You should use dataset.take(k).cache().repeat() instead. (linenv) sunlab@sunlab-ThinkStation-P520:~/superpoint/SuperPoint-master/superpoint$ python experiment.py train configs/magic-point_shapes.yaml magic-point_synth

00ohnim commented 1 year ago

I ran into this problem,too,but I don't know how to solve it.

1z2213 commented 1 year ago

Add heading textAdd bold text, <Ctrl+b>Add italic text, <Ctrl+i> Add a quote, <Ctrl+Shift+.>Add code, <Ctrl+e>Add a link, <Ctrl+k> Add a bulleted list, <Ctrl+Shift+8>Add a numbered list, <Ctrl+Shift+7>Add a task list, <Ctrl+Shift+l> Directly mention a user or team Reference an issue, pull request, or discussion Add saved reply Slash commands It seems like a lot of people experienced this problem, but those who experienced it don't seem to have a solution for it.

rpautrat commented 1 year ago

Hi, this might be due to the warning about the cache returned by tensorflow at the end of your output. Could you try training without caching the data, i.e. by setting 'cache_in_memory' to false in superpoint/configs/magic-point_shapes.yaml?

Alternatively, you could also add the field 'on-the-fly' in a new line after 'cache_in_memory' and set it to true. This would generate the shapes on the fly instead of pre-generating them, and could be a good way to check if the error is caused by caching issues.

1z2213 commented 1 year ago

Hi, this might be due to the warning about the cache returned by tensorflow at the end of your output. Could you try training without caching the data, i.e. by setting 'cache_in_memory' to false in superpoint/configs/magic-point_shapes.yaml?

Alternatively, you could also add the field 'on-the-fly' in a new line after 'cache_in_memory' and set it to true. This would generate the shapes on the fly instead of pre-generating them, and could be a good way to check if the error is caused by caching issues.

I apologize for responding to your message so late, I tried your method but it didn't solve my problem. This proves that the error is not caused by caching issues.