affinelayer / pix2pix-tensorflow

Tensorflow port of Image-to-Image Translation with Conditional Adversarial Nets https://phillipi.github.io/pix2pix/
MIT License
5.07k stars 1.3k forks source link

Invalid argument: During Variant Host->Device Copy: non-DMA-copy attempted of tensor type: string #188

Closed burhr2 closed 4 years ago

burhr2 commented 4 years ago

Hi, I upgraded the codes to TensorFlow 2 using the guide from [this post] (https://medium.com/tensorflow/upgrading-your-code-to-tensorflow-2-0-f72c3a4d83b5) and managed to train the model one time on a custom dataset. After I tried to re-run I am encountering the error mentioned. I tried to read from this post but couldn't understand how to solve this

  1. Using TensorFlow-GPU v 2.1.0 with anaconda
  2. Spyder ide

` File "C:\ProgramData\Anaconda3\envs\TF_GPU\lib\site-packages\tensorflow_core\python\client\session.py", line 1445, in _call_tf_sessionrun run_metadata)

InvalidArgumentError: 2 root error(s) found. (0) Invalid argument: 2 root error(s) found. (0) Invalid argument: During Variant Host->Device Copy: non-DMA-copy attempted of tensor type: string (1) Invalid argument: During Variant Host->Device Copy: non-DMA-copy attempted of tensor type: string 0 successful operations. 0 derived errors ignored. [[{{node TensorArrayV2Write/TensorListSetItem}}]] (1) Invalid argument: 2 root error(s) found. (0) Invalid argument: During Variant Host->Device Copy: non-DMA-copy attempted of tensor type: string (1) Invalid argument: During Variant Host->Device Copy: non-DMA-copy attempted of tensor type: string 0 successful operations. 0 derived errors ignored. [[{{node TensorArrayV2Write/TensorListSetItem}}]] [[encode_images/target_pngs/while/exit/_68/_697]] 0 successful operations. 0 derived errors ignored.`

burhr2 commented 4 years ago

Hi, for anyone who may encounter this error in future. I tried to follow the comment from this

changed a few lines in pix2pix from

` with tf.compat.v1.name_scope("encode_images"):
        display_fetches = {
            "paths": examples.paths,
            "inputs": tf.map_fn(tf.image.encode_png, converted_inputs, dtype=tf.string, name="input_pngs"),
            "targets": tf.map_fn(tf.image.encode_png, converted_targets, dtype=tf.string, name="target_pngs"),
            "outputs": tf.map_fn(tf.image.encode_png, converted_outputs, dtype=tf.string, name="output_pngs"),
        }` 

To

`    with tf.compat.v1.name_scope("encode_images"):
        with tf.device("/cpu:0"):
            display_fetches = {
                "paths": examples.paths,
                "inputs": tf.map_fn(tf.image.encode_png, converted_inputs, dtype=tf.string, name="input_pngs"),
                "targets": tf.map_fn(tf.image.encode_png, converted_targets, dtype=tf.string, name="target_pngs"),
                "outputs": tf.map_fn(tf.image.encode_png, converted_outputs, dtype=tf.string, name="output_pngs"),
            }
`

Although this i a temporary solution