dementrock
commented
7 years ago This seems like a Theano-specific issue. Maybe ask for help on their github repo or the google group? https://groups.google.com/forum/#!forum/theano-users
Closed xquan999 closed 7 years ago
dementrock
commented
7 years ago This seems like a Theano-specific issue. Maybe ask for help on their github repo or the google group? https://groups.google.com/forum/#!forum/theano-users
xquan999
commented
7 years ago Thanks, I will try
When running with Theano and cuda , some thing seems wrong
` (rllab3) xquan@xquan-Desktop:~/workspace/DL/rllab/examples$ python trpo_gym_pendulum.py
Using cuDNN version 5105 on context None Mapped name None to device cuda: GeForce GTX 1070 (0000:01:00.0) python /home/xquan/workspace/DL/rllab/scripts/run_experiment_lite.py --seed '1' --variant_data 'gAN9cQBYCAAAAGV4cF9uYW1lcQFYIwAAAGV4cGVyaW1lbnRfMjAxN18wN18yMF8wM18zMF80NF8wMDAxcQJzLg==' --log_dir '/home/xquan/workspace/DL/rllab/data/local/experiment/experiment_2017_07_20_03_30_44_0001' --n_parallel '1' --snapshot_mode 'last' --exp_name 'experiment_2017_07_20_03_30_44_0001' --use_cloudpickle 'True' --args_data 'gAJjY2xvdWRwaWNrbGUuY2xvdWRwaWNrbGUKX2ZpbGxfZnVuY3Rpb24KcQAoY2Nsb3VkcGlja2xlLmNsb3VkcGlja2xlCl9tYWtlX3NrZWxfZnVuYwpxAWNjbG91ZHBpY2tsZS5jbG91ZHBpY2tsZQpfYnVpbHRpbl90eXBlCnECWAgAAABDb2RlVHlwZXEDhXEEUnEFKEsASwBLBUsTS0djX2NvZGVjcwplbmNvZGUKcQZYkgAAAHQAAHQBAGQBAMKDAQDCgwEAfQEAdAIAZAIAfAEAagMAZAMAZBMAwoMAAn0CAHQEAGQCAHwBAGoDAMKDAAF9AwB0BQBkBQB8AQBkBgB8AgBkBwB8AwBkCABkCQBkCgB8AQBqBgBkCwBkDABkDQBkDgBkDwBkEABkEQBkEgDCgwAJfQQAfAQAagcAwoMAAAFkAABTcQdYBgAAAGxhdGluMXEIhnEJUnEKKE5YCwAAAFBlbmR1bHVtLXYwcQtYCAAAAGVudl9zcGVjcQxYDAAAAGhpZGRlbl9zaXplc3ENSyBYAwAAAGVudnEOWAYAAABwb2xpY3lxD1gIAAAAYmFzZWxpbmVxEFgKAAAAYmF0Y2hfc2l6ZXERTaAPWA8AAABtYXhfcGF0aF9sZW5ndGhxElgFAAAAbl9pdHJxE0syWAgAAABkaXNjb3VudHEURz/vrhR64UeuWAkAAABzdGVwX3NpemVxFUc/hHrhR64Ue1gEAAAAcGxvdHEWiEsgSyCGcRd0cRgoWAkAAABub3JtYWxpemVxGVgGAAAAR3ltRW52cRpYEQAAAEdhdXNzaWFuTUxQUG9saWN5cRtYBAAAAHNwZWNxHFgVAAAATGluZWFyRmVhdHVyZUJhc2VsaW5lcR1YBAAAAFRSUE9xHlgHAAAAaG9yaXpvbnEfWAUAAAB0cmFpbnEgdHEhKFgBAAAAX3EiaA5oD2gQWAQAAABhbGdvcSN0cSRYFAAAAHRycG9fZ3ltX3BlbmR1bHVtLnB5cSVYCAAAAHJ1bl90YXNrcSZLCWgGWCAAAAAABBICBgEJAgkDEgIGAQYBBgEGAQYBCQEGAQYBBgIJAnEnaAiGcShScSkpKXRxKlJxK0r/////fXEsh3EtUnEufXEvKGgbY3JsbGFiLnBvbGljaWVzLmdhdXNzaWFuX21scF9wb2xpY3kKR2F1c3NpYW5NTFBQb2xpY3kKcTBoGWNybGxhYi5lbnZzLm5vcm1hbGl6ZWRfZW52Ck5vcm1hbGl6ZWRFbnYKcTFoHmNybGxhYi5hbGdvcy50cnBvClRSUE8KcTJoHWNybGxhYi5iYXNlbGluZXMubGluZWFyX2ZlYXR1cmVfYmFzZWxpbmUKTGluZWFyRmVhdHVyZUJhc2VsaW5lCnEzaBpjcmxsYWIuZW52cy5neW1fZW52Ckd5bUVudgpxNHVOfXE1TnRSLg==' --plot 'True' Using cuDNN version 5105 on context None Mapped name None to device cuda: GeForce GTX 1070 (0000:01:00.0) using seed 1 2017-07-20 03:30:50.001703 CST | Setting seed to 1 using seed 1 [2017-07-20 03:30:50,168] Making new env: Pendulum-v0 2017-07-20 03:30:50.173841 CST | [experiment_2017_07_20_03_30_44_0001] observation space: Box(3,) 2017-07-20 03:30:50.174652 CST | [experiment_2017_07_20_03_30_44_0001] action space: Box(1,) 2017-07-20 03:30:58.554635 CST | [experiment_2017_07_20_03_30_44_0001] Populating workers... 2017-07-20 03:30:58.554883 CST | [experiment_2017_07_20_03_30_44_0001] Populated Traceback (most recent call last): File "/home/xquan/anaconda3/envs/rllab3/lib/python3.5/multiprocessing/queues.py", line 241, in _feed obj = ForkingPickler.dumps(obj) File "/home/xquan/anaconda3/envs/rllab3/lib/python3.5/multiprocessing/reduction.py", line 50, in dumps cls(buf, protocol).dump(obj) File "/home/xquan/workspace/DL/rllab/rllab/core/parameterized.py", line 77, in getstate d["params"] = self.get_param_values() File "/home/xquan/workspace/DL/rllab/rllab/core/parameterized.py", line 57, in get_param_values for param in self.get_params(tags)] File "/home/xquan/workspace/DL/rllab/rllab/core/parameterized.py", line 57, in
for param in self.get_params( tags)]
File "/home/xquan/.local/lib/python3.5/site-packages/theano/gpuarray/type.py", line 602, in get_value
return np.asarray(self.container.value)
File "/home/xquan/.local/lib/python3.5/site-packages/numpy/core/numeric.py", line 531, in asarray
return array(a, dtype, copy=False, order=order)
File "pygpu/gpuarray.pyx", line 1734, in pygpu.gpuarray.GpuArray.array (pygpu/gpuarray.c:21586)
File "pygpu/gpuarray.pyx", line 1407, in pygpu.gpuarray._pygpu_as_ndarray (pygpu/gpuarray.c:18268)
File "pygpu/gpuarray.pyx", line 394, in pygpu.gpuarray.array_read (pygpu/gpuarray.c:6932)
pygpu.gpuarray.GpuArrayException: b'cuMemcpyDtoHAsync(dst, src->ptr + srcoff, sz, ctx->mem_s): CUDA_ERROR_INVALID_VALUE: invalid argument'
[2017-07-20 03:30:58,665] Starting new video recorder writing to /home/xquan/workspace/DL/rllab/data/local/experiment/experiment_2017_07_20_03_30_44_0001/gym_log/openaigym.video.0.7592.video000000.mp4
0% [# ] 100% | ETA: 00:02:26[2017-07-20 03:31:06,561] Starting new video recorder writing to /home/xquan/workspace/DL/rllab/data/local/experiment/experiment_2017_07_20_03_30_44_0001/gym_log/openaigym.video.0.7592.video000001.mp4
0% [############ ] 100% | ETA: 00:00:22[2017-07-20 03:31:13,861] Starting new video recorder writing to /home/xquan/workspace/DL/rllab/data/local/experiment/experiment_2017_07_20_03_30_44_0001/gym_log/openaigym.video.0.7592.video000008.mp4
0% [##############################] 100% | ETA: 00:00:00
Total time elapsed: 00:00:22
2017-07-20 03:31:21.473784 CST | [experiment_2017_07_20_03_30_44_0001] itr #0 | fitting baseline...
2017-07-20 03:31:21.492058 CST | [experiment_2017_07_20_03_30_44_0001] itr #0 | fitted
=: Compiling function f_loss
Segmentation fault (core dumped)
`
Here is my .theanorc configuration ` [global] device=cuda floatX=float32
[cuda] root=/usr/local/cuda-8.0
[lib] cnmem=1 `
What should I do ?