sudo docker run --rm --gpus all nvidia/cuda:11.0-base nvidia-smi
Mon Jan 18 18:07:24 2021
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 460.32.03 Driver Version: 460.32.03 CUDA Version: 11.2 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|===============================+======================+======================|
| 0 GeForce GTX 105... Off | 00000000:01:00.0 Off | N/A |
| N/A 47C P8 N/A / N/A | 419MiB / 4040MiB | 0% Default |
| | | N/A |
+-------------------------------+----------------------+----------------------+
+-----------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=============================================================================|
+-----------------------------------------------------------------------------+
错误发生在 convert to kmodel fail
2021-01-18 18:01:02.319213: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1241] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 3229 MB memory) -> physical GPU (device: 0, name: GeForce GTX 1050 Ti, pci bus id: 0000:01:00.0, compute capability: 6.1)
2021-01-18 18:01:02.370916: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:814] Optimization results for grappler item: graph_to_optimize
2021-01-18 18:01:02.370945: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:816] constant_folding: Graph size after: 234 nodes (-137), 370 edges (-137), time = 28.504ms.
2021-01-18 18:01:02.370973: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:816] constant_folding: Graph size after: 234 nodes (0), 370 edges (0), time = 11.414ms.
[ERROR] -6 b'' b"FailFast:\nCouldn't find a valid ICU package installed on the system. Set the configuration flag System.Globalization.Invariant to true if you want to run with no globalization support.\n\n at System.Environment.FailFast(System.String)\n at System.Globalization.GlobalizationMode.GetGlobalizationInvariantMode()\n at System.Globalization.GlobalizationMode..cctor()\n at System.Globalization.CultureData.CreateCultureWithInvariantData()\n at System.Globalization.CultureData.get_Invariant()\n at System.Globalization.CultureData.GetCultureData(System.String, Boolean)\n at System.Globalization.CultureInfo..ctor(System.String, Boolean)\n at System.Reflection.RuntimeAssembly.GetLocale()\n at System.Reflection.RuntimeAssembly.GetName(Boolean)\n at System.Reflection.Assembly.GetName()\n at System.Diagnostics.Tracing.EventPipeController.GetAppName()\n at System.Diagnostics.Tracing.EventPipeController..ctor()\n at System.Diagnostics.Tracing.EventPipeController.Initialize()\n at System.StartupHookProvider.ProcessStartupHooks()\n"
2021-01-18 18:01:04,682 - [ERROR]: convert to kmodel fail
2021-01-18 18:01:04,683 - [ERROR]: failed: TrainFailReason.ERROR_INTERNAL, convert kmodel fail: output:
b''
error:
b"FailFast:\nCouldn't find a valid ICU package installed on the system. Set the configuration flag System.Globalization.Invariant to true if you want to run with no globalization support.\n\n at System.Environment.FailFast(System.String)\n at System.Globalization.GlobalizationMode.GetGlobalizationInvariantMode()\n at System.Globalization.GlobalizationMode..cctor()\n at System.Globalization.CultureData.CreateCultureWithInvariantData()\n at System.Globalization.CultureData.get_Invariant()\n at System.Globalization.CultureData.GetCultureData(System.String, Boolean)\n at System.Globalization.CultureInfo..ctor(System.String, Boolean)\n at System.Reflection.RuntimeAssembly.GetLocale()\n at System.Reflection.RuntimeAssembly.GetName(Boolean)\n at System.Reflection.Assembly.GetName()\n at System.Diagnostics.Tracing.EventPipeController.GetAppName()\n at System.Diagnostics.Tracing.EventPipeController..ctor()\n at System.Diagnostics.Tracing.EventPipeController.Initialize()\n at System.StartupHookProvider.ProcessStartupHooks()\n"
训练出错,请帮忙看一下。谢谢 Docker,GPU情况如下:
sudo docker run --rm --gpus all nvidia/cuda:11.0-base nvidia-smi Mon Jan 18 18:07:24 2021 +-----------------------------------------------------------------------------+ | NVIDIA-SMI 460.32.03 Driver Version: 460.32.03 CUDA Version: 11.2 | |-------------------------------+----------------------+----------------------+ | GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC | | Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. | | | | MIG M. | |===============================+======================+======================| | 0 GeForce GTX 105... Off | 00000000:01:00.0 Off | N/A | | N/A 47C P8 N/A / N/A | 419MiB / 4040MiB | 0% Default | | | | N/A | +-------------------------------+----------------------+----------------------+
+-----------------------------------------------------------------------------+ | Processes: | | GPU GI CI PID Type Process name GPU Memory | | ID ID Usage | |=============================================================================| +-----------------------------------------------------------------------------+
错误发生在 convert to kmodel fail
2021-01-18 18:01:02.319213: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1241] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 3229 MB memory) -> physical GPU (device: 0, name: GeForce GTX 1050 Ti, pci bus id: 0000:01:00.0, compute capability: 6.1) 2021-01-18 18:01:02.370916: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:814] Optimization results for grappler item: graph_to_optimize 2021-01-18 18:01:02.370945: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:816] constant_folding: Graph size after: 234 nodes (-137), 370 edges (-137), time = 28.504ms. 2021-01-18 18:01:02.370973: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:816] constant_folding: Graph size after: 234 nodes (0), 370 edges (0), time = 11.414ms. [ERROR] -6 b'' b"FailFast:\nCouldn't find a valid ICU package installed on the system. Set the configuration flag System.Globalization.Invariant to true if you want to run with no globalization support.\n\n at System.Environment.FailFast(System.String)\n at System.Globalization.GlobalizationMode.GetGlobalizationInvariantMode()\n at System.Globalization.GlobalizationMode..cctor()\n at System.Globalization.CultureData.CreateCultureWithInvariantData()\n at System.Globalization.CultureData.get_Invariant()\n at System.Globalization.CultureData.GetCultureData(System.String, Boolean)\n at System.Globalization.CultureInfo..ctor(System.String, Boolean)\n at System.Reflection.RuntimeAssembly.GetLocale()\n at System.Reflection.RuntimeAssembly.GetName(Boolean)\n at System.Reflection.Assembly.GetName()\n at System.Diagnostics.Tracing.EventPipeController.GetAppName()\n at System.Diagnostics.Tracing.EventPipeController..ctor()\n at System.Diagnostics.Tracing.EventPipeController.Initialize()\n at System.StartupHookProvider.ProcessStartupHooks()\n" 2021-01-18 18:01:04,682 - [ERROR]: convert to kmodel fail 2021-01-18 18:01:04,683 - [ERROR]: failed: TrainFailReason.ERROR_INTERNAL, convert kmodel fail: output: b'' error: b"FailFast:\nCouldn't find a valid ICU package installed on the system. Set the configuration flag System.Globalization.Invariant to true if you want to run with no globalization support.\n\n at System.Environment.FailFast(System.String)\n at System.Globalization.GlobalizationMode.GetGlobalizationInvariantMode()\n at System.Globalization.GlobalizationMode..cctor()\n at System.Globalization.CultureData.CreateCultureWithInvariantData()\n at System.Globalization.CultureData.get_Invariant()\n at System.Globalization.CultureData.GetCultureData(System.String, Boolean)\n at System.Globalization.CultureInfo..ctor(System.String, Boolean)\n at System.Reflection.RuntimeAssembly.GetLocale()\n at System.Reflection.RuntimeAssembly.GetName(Boolean)\n at System.Reflection.Assembly.GetName()\n at System.Diagnostics.Tracing.EventPipeController.GetAppName()\n at System.Diagnostics.Tracing.EventPipeController..ctor()\n at System.Diagnostics.Tracing.EventPipeController.Initialize()\n at System.StartupHookProvider.ProcessStartupHooks()\n"