wangwei-bee
commented
2 years ago 8
D:\ProgramData\Anaconda3\envs\tf1\python.exe D:/env/seq2point-nilm-master/train_main.py
2021-12-07 18:39:29.084196: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cudart64_110.dll
2021-12-07 18:39:33.692646: I tensorflow/compiler/jit/xla_cpu_device.cc:41] Not creating XLA devices, tf_xla_enable_xla_devices not set
2021-12-07 18:39:33.693571: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library nvcuda.dll
2021-12-07 18:39:33.734057: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties:
pciBusID: 0000:73:00.0 name: GeForce RTX 3090 computeCapability: 8.6
coreClock: 1.695GHz coreCount: 82 deviceMemorySize: 24.00GiB deviceMemoryBandwidth: 871.81GiB/s
2021-12-07 18:39:33.734245: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cudart64_110.dll
2021-12-07 18:39:33.756459: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cublas64_11.dll
2021-12-07 18:39:33.756564: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cublasLt64_11.dll
2021-12-07 18:39:33.761675: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cufft64_10.dll
2021-12-07 18:39:33.763061: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library curand64_10.dll
2021-12-07 18:39:33.774396: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cusolver64_10.dll
2021-12-07 18:39:33.778510: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cusparse64_11.dll
2021-12-07 18:39:33.779184: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cudnn64_8.dll
2021-12-07 18:39:33.779337: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1862] Adding visible gpu devices: 0
2021-12-07 18:39:33.779650: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations: AVX2
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2021-12-07 18:39:33.786034: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties:
pciBusID: 0000:73:00.0 name: GeForce RTX 3090 computeCapability: 8.6
coreClock: 1.695GHz coreCount: 82 deviceMemorySize: 24.00GiB deviceMemoryBandwidth: 871.81GiB/s
2021-12-07 18:39:33.786300: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cudart64_110.dll
2021-12-07 18:39:33.786434: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cublas64_11.dll
2021-12-07 18:39:33.786566: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cublasLt64_11.dll
2021-12-07 18:39:33.786698: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cufft64_10.dll
2021-12-07 18:39:33.786824: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library curand64_10.dll
2021-12-07 18:39:33.786960: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cusolver64_10.dll
2021-12-07 18:39:33.787080: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cusparse64_11.dll
2021-12-07 18:39:33.787196: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cudnn64_8.dll
2021-12-07 18:39:33.787338: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1862] Adding visible gpu devices: 0
2021-12-07 18:39:34.533384: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1261] Device interconnect StreamExecutor with strength 1 edge matrix:
2021-12-07 18:39:34.533500: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1267] 0
2021-12-07 18:39:34.533560: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1280] 0: N
2021-12-07 18:39:34.533799: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1406] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 21821 MB memory) -> physical GPU (device: 0, name: GeForce RTX 3090, pci bus id: 0000:73:00.0, compute capability: 8.6)
2021-12-07 18:39:34.535316: I tensorflow/compiler/jit/xla_gpu_device.cc:99] Not creating XLA devices, tf_xla_enable_xla_devices not set
Importing training file...
Counting number of rows...
Done.
The dataset contains 300000 rows
path is : D:/env/seq2point-nilm-master/dataset_management/redd/REDD1/dishwashertraining.csv
aggregate dishwasher
0 -0.594218 -0.699000
1 -0.594218 -0.699000
2 -0.594211 -0.699667
3 -0.594240 -0.699000
4 -0.594295 -0.699000
... ... ...
299995 -0.059237 -0.700000
299996 -0.055958 -0.700000
299997 -0.045611 -0.699500
299998 -0.052087 -0.700000
299999 -0.049189 -0.700000
[300000 rows x 2 columns] (300000,) 2021-12-07 18:39:36.360595: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:116] None of the MLIR optimization passes are enabled (registered 2) Epoch 1/50 2021-12-07 18:39:36.836693: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cublas64_11.dll 2021-12-07 18:39:37.737411: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cublasLt64_11.dll 2021-12-07 18:39:37.769089: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cudnn64_8.dll 2021-12-07 18:39:39.559999: I tensorflow/core/platform/windows/subprocess.cc:308] SubProcess ended with return code: 0
2021-12-07 18:39:39.641183: I tensorflow/core/platform/windows/subprocess.cc:308] SubProcess ended with return code: 0
2021-12-07 18:39:39.839037: I tensorflow/stream_executor/cuda/cuda_blas.cc:1838] TensorFloat-32 will be used for the matrix multiplication. This will only be logged once. 585/585 [==============================] - ETA: 0s - loss: 0.0193 - mse: 0.0193 - msle: 5.0212e-04 - mae: 0.0561Importing training file... Counting number of rows... Done. The dataset contains 33372 rows path is : D:/env/seq2point-nilm-master/dataset_management/redd/REDD1/dishwashervalidation.csv aggregate dishwasher 0 -0.284920 -0.6990 1 -0.284994 -0.6990 2 -0.282996 -0.6990 3 -0.282336 -0.6990 4 -0.284320 -0.6990 ... ... ... 33367 -0.619515 -0.6970 33368 -0.619501 -0.6975 33369 -0.619524 -0.6970 33370 -0.619579 -0.6970 33371 -0.619562 -0.6970
[33372 rows x 2 columns]
(33372,)
585/585 [==============================] - 48s 75ms/step - loss: 0.0193 - mse: 0.0193 - msle: 5.0214e-04 - mae: 0.0561 - val_loss: 0.0048 - val_mse: 0.0048 - val_msle: 1.2839e-05 - val_mae: 0.0179
Epoch 2/50
64/585 [==>...........................] - ETA: 36s - loss: 0.0023 - mse: 0.0023 - msle: 8.2943e-05 - mae: 0.01982021-12-07 18:40:28.745127: W tensorflow/core/framework/op_kernel.cc:1751] Invalid argument: ValueError: generator yielded an element of shape (0,) where an element of shape (None, None) was expected.
Traceback (most recent call last):
File "D:\ProgramData\Anaconda3\envs\tf1\lib\site-packages\tensorflow\python\ops\script_ops.py", line 249, in call ret = func(*args)
File "D:\ProgramData\Anaconda3\envs\tf1\lib\site-packages\tensorflow\python\autograph\impl\api.py", line 620, in wrapper return func(*args, **kwargs)
File "D:\ProgramData\Anaconda3\envs\tf1\lib\site-packages\tensorflow\python\data\ops\dataset_ops.py", line 938, in generator_py_func (ret_array.shape, expected_shape))
ValueError: generator yielded an element of shape (0,) where an element of shape (None, None) was expected.
Traceback (most recent call last):
File "D:/env/seq2point-nilm-master/train_main.py", line 42, in generator yielded an element of shape (0,) where an element training_history = self.default_train(model, callbacks, steps_per_training_epoch)
of sh File "D:\env\seq2point-nilm-master\seq2point_train.py", line 162, in default_train
ape (None, None) was expected.
Traceback (most recent call last):
File "D:\ProgramData\Anaconda3\envs\tf1\lib\site-packages\tensorflow\python\ops\script_ops.py", line 249, in call ret = func(*args) validation_steps=self.__validation_steps)
Fi File "D:\ProgramData\Anaconda3\envs\tf1\lib\site-packages\tensorflow\python\keras\engine\training.py", line 1100, in fit le "D:\ProgramData\Anaconda3\envs\tf1\lib\site-packages\tensorflow\python\autograph\impl\api.py", line 620, in wrapper return func(*args, **kwargs)
File "D:\ProgramData\Anaconda3\envs\tf1\lib\site-packages\tensorflow\python\data\ops\dataset_ops.py", line 938, in generator_py_func (ret_array.shape, expected_shape))
ValueError: generator yielded an element of shape (0,) where an element of shape (None, None) was expected.
tmp_logs = self.train_function(iterator)File "D:\ProgramData\Anaconda3\envs\tf1\lib\site-packages\tensorflow\python\eager\def_function.py", line 828, in call
result = self._call(*args, *kwds)
File "D:\ProgramData\Anaconda3\envs\tf1\lib\site-packages\tensorflow\python\eager\def_function.py", line 855, in _call
return self._stateless_fn(args, **kwds) # pylint: disable=not-callable
File "D:\ProgramData\Anaconda3\envs\tf1\lib\site-packages\tensorflow\python\eager\function.py", line 2943, in call
filtered_flat_args, captured_inputs=graph_function.captured_inputs) # pylint: disable=protected-access
File "D:\ProgramData\Anaconda3\envs\tf1\lib\site-packages\tensorflow\python\eager\function.py", line 1919, in _call_flat
ctx, args, cancellation_manager=cancellation_manager))
File "D:\ProgramData\Anaconda3\envs\tf1\lib\site-packages\tensorflow\python\eager\function.py", line 560, in call
ctx=ctx)
File "D:\ProgramData\Anaconda3\envs\tf1\lib\site-packages\tensorflow\python\eager\execute.py", line 60, in quick_execute
inputs, attrs, num_outputs)
tensorflow.python.framework.errors_impl.InvalidArgumentError: 2 root error(s) found.
(0) Invalid argument: ValueError: generator yielded an element of shape (0,) where an element of shape (None, None) was expected.
Traceback (most recent call last):
File "D:\ProgramData\Anaconda3\envs\tf1\lib\site-packages\tensorflow\python\ops\script_ops.py", line 249, in call ret = func(*args)
File "D:\ProgramData\Anaconda3\envs\tf1\lib\site-packages\tensorflow\python\autograph\impl\api.py", line 620, in wrapper return func(*args, **kwargs)
File "D:\ProgramData\Anaconda3\envs\tf1\lib\site-packages\tensorflow\python\data\ops\dataset_ops.py", line 938, in generator_py_func (ret_array.shape, expected_shape))
ValueError: generator yielded an element of shape (0,) where an element of shape (None, None) was expected.
[[{{node PyFunc}}]]
[[IteratorGetNext]](1) Invalid argument: ValueError: generator yielded an element of shape (0,) where an element of shape (None, None) was expected.
Traceback (most recent call last):
File "D:\ProgramData\Anaconda3\envs\tf1\lib\site-packages\tensorflow\python\ops\script_ops.py", line 249, in call ret = func(*args)
File "D:\ProgramData\Anaconda3\envs\tf1\lib\site-packages\tensorflow\python\autograph\impl\api.py", line 620, in wrapper return func(*args, **kwargs)
File "D:\ProgramData\Anaconda3\envs\tf1\lib\site-packages\tensorflow\python\data\ops\dataset_ops.py", line 938, in generator_py_func (ret_array.shape, expected_shape))
ValueError: generator yielded an element of shape (0,) where an element of shape (None, None) was expected.
[[{{node PyFunc}}]]
[[IteratorGetNext]]
[[IteratorGetNext/_10]]0 successful operations. 0 derived errors ignored. [Op:__inference_train_function_1084]
Function call stack: train_function -> train_function
2021-12-07 18:40:28.958495: W tensorflow/core/kernels/data/generator_dataset_op.cc:107] Error occurred when finalizing GeneratorDataset iterator: Failed precondition: Python interpreter state is not initialized. The process may be terminated. [[{{node PyFunc}}]]
Process finished with exit code 1
MingjunZhong
I don't understand your question.
Originally posted by @MingjunZhong in https://github.com/MingjunZhong/seq2point-nilm/issues/8#issuecomment-987753862