search

kbrodt / waymo-motion-prediction-2021

Waymo: Motion Prediction 2021
186 stars 45 forks source link

Preprocessing issue #8

Open karanveersingh5623 opened 2 years ago

karanveersingh5623 commented 2 years ago

I tried running with all requirements , it keeps on reading the records but when it tries to write down .....here it fails , please help me to proceed further

root@6f4483fc53c1:/app/waymo-adas-main/waymo-motion-prediction-2021# python3 prerender.py --data /app/waymo-adas-main/waymo-dataset/original/validation/ --out /app/waymo-adas-main/data/train1
False
Namespace(data='/app/waymo-adas-main/waymo-dataset/original/validation/', each=0, n_jobs=20, n_shards=8, no_valid=False, out='/app/waymo-adas-main/data/train1', use_vectorize=False)
1215it [00:28, 42.50it/s]
  0%|                                                                                                                                                                                   | 0/1215 [00:00<?, ?it/s]
multiprocessing.pool.RemoteTraceback:
"""
Traceback (most recent call last):
  File "/usr/lib/python3.8/multiprocessing/pool.py", line 125, in worker
    result = (True, func(*args, **kwds))
  File "prerender.py", line 731, in merge
    parsed = tf.io.parse_single_example(data, features_description)
  File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/util/traceback_utils.py", line 153, in error_handler
    raise e.with_traceback(filtered_tb) from None
  File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/execute.py", line 58, in quick_execute
    tensors = pywrap_tfe.TFE_Py_Execute(ctx._handle, device_name, op_name,
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x96 in position 40: invalid start byte
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "prerender.py", line 836, in <module>
    main()
  File "prerender.py", line 832, in main
    r.get()
  File "/usr/lib/python3.8/multiprocessing/pool.py", line 771, in get
    raise self._value
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x96 in position 40: invalid start byte
kbrodt commented 2 years ago

Try to remove multiprocessing and debug

karanveersingh5623 commented 2 years ago

Changed the code , got the same error

#    p = multiprocessing.Pool(args.n_jobs)
    proc_id = 0
#    res = []
    for data in tqdm(dataset.as_numpy_iterator()):
#        print(data)
        proc_id += 1
        kwds=dict(data=data, proc_id=proc_id, validate=not args.no_valid, out_dir=args.out, use_vectorize=args.use_vectorize)
#        print(kwds)
        merge(**kwds)

#    for r in tqdm(res):
#        r.get()

if __name__ == "__main__":
    main()

OUTPUT

root@6f4483fc53c1:/app/waymo-adas-main/waymo-motion-prediction-2021# python3 prerender1.py --data /app/waymo-adas-main/waymo-dataset/original/train/ --out /app/waymo-adas-main/data/train1
False
Namespace(data='/app/waymo-adas-main/waymo-dataset/original/train/', each=0, n_jobs=1, n_shards=8, no_valid=False, out='/app/waymo-adas-main/data/train1', use_vectorize=False)
<ShardDataset shapes: (), types: tf.string>
0it [00:00, ?it/s]
Traceback (most recent call last):
  File "prerender1.py", line 829, in <module>
    main()
  File "prerender1.py", line 822, in main
    merge(**kwds)
  File "prerender1.py", line 731, in merge
    parsed = tf.io.parse_single_example(data, features_description)
  File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/util/traceback_utils.py", line 153, in error_handler
    raise e.with_traceback(filtered_tb) from None
  File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/execute.py", line 58, in quick_execute
    tensors = pywrap_tfe.TFE_Py_Execute(ctx._handle, device_name, op_name,
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x96 in position 40: invalid start byte
kbrodt commented 2 years ago

It's a problem of tensorflow. Are data and features_description in parsed = tf.io.parse_single_example(data, features_description) correct?

karanveersingh5623 commented 2 years ago

Thanks for reply I just used one tf record with above code , it also failed . So data looks fine Coming to features description , its defined in the script , so I couldn't figure out . What else I can try ?

features_description = {}
features_description.update(roadgraph_features)
features_description.update(state_features)
features_description.update(traffic_light_features)
kbrodt commented 2 years ago

I've tested on my machine and there were no errors.

The tensorflow version:

tensorflow==2.5.0
tensorflow-addons==0.10.0
tensorflow-estimator==1.13.0
tensorflow-gpu==2.0.0
karanveersingh5623 commented 2 years ago

Same Error , below are the versions information . I am using Tensorflow 2.5.0 GPU docker image .

root@d9fa28761b2f:/app/waymo-adas-main/waymo-motion-prediction-2021# python3 prerender1.py --data /app/waymo-adas-main/waymo-dataset/original/train/ --out /app/waymo-adas-main/data/train1
Namespace(data='/app/waymo-adas-main/waymo-dataset/original/train/', each=0, n_jobs=1, n_shards=8, no_valid=False, out='/app/waymo-adas-main/data/train1', use_vectorize=False)
2021-12-13 04:53:46.102202: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcuda.so.1
2021-12-13 04:53:46.102231: E tensorflow/stream_executor/cuda/cuda_driver.cc:318] failed call to cuInit: UNKNOWN ERROR (34)
2021-12-13 04:53:46.102255: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:163] no NVIDIA GPU device is present: /dev/nvidia0 does not exist
2021-12-13 04:53:46.102490: I tensorflow/core/platform/cpu_feature_guard.cc:142] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 AVX512F FMA
2021-12-13 04:53:46.141731: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 2100000000 Hz
2021-12-13 04:53:46.157862: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x476d4d0 executing computations on platform Host. Devices:
2021-12-13 04:53:46.157922: I tensorflow/compiler/xla/service/service.cc:175]   StreamExecutor device (0): Host, Default Version
<ShardDataset shapes: (), types: tf.string>
0it [00:00, ?it/s]2021-12-13 04:53:46.248422: W tensorflow/core/framework/op_kernel.cc:1622] OP_REQUIRES failed at example_parsing_ops.cc:240 : Invalid argument: Could not parse example input, value: '

&17642771458376274038_2080_000_2100_009@9@E@`l-;@s?ֿfP?EH9#-3
0it [00:00, ?it/s]
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/ops/gen_parsing_ops.py", line 1155, in parse_single_example
    "dense_shapes", dense_shapes)
tensorflow.python.eager.core._FallbackException: Expecting None or sequence value for element0 of attr dense_shapes, got TensorShapeProto

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "prerender1.py", line 830, in <module>
    main()
  File "prerender1.py", line 823, in main
    merge(**kwds)
  File "prerender1.py", line 731, in merge
    parsed = tf.io.parse_single_example(data, features_description)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/ops/parsing_ops.py", line 1063, in parse_single_example_v2_unoptimized
    return parse_single_example_v2(serialized, features, name)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/ops/parsing_ops.py", line 2093, in parse_single_example_v2
    dense_defaults, dense_shapes, name)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/ops/parsing_ops.py", line 2210, in _parse_single_example_v2_raw
    name=name)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/ops/gen_parsing_ops.py", line 1164, in parse_single_example
    ctx=_ctx)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/ops/gen_parsing_ops.py", line 1260, in parse_single_example_eager_fallback
    attrs=_attrs, ctx=_ctx, name=name)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/eager/execute.py", line 61, in quick_execute
    num_outputs)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x96 in position 40: invalid start byte

Packages Versions

root@d9fa28761b2f:/app/waymo-adas-main/waymo-motion-prediction-2021# python3 -m pip show tensorflow
Name: tensorflow
Version: 2.5.0
Summary: TensorFlow is an open source machine learning framework for everyone.
Home-page: https://www.tensorflow.org/
Author: Google Inc.
Author-email: packages@tensorflow.org
License: Apache 2.0
Location: /usr/local/lib/python3.6/dist-packages
Requires: absl-py, astunparse, flatbuffers, gast, google-pasta, grpcio, h5py, keras-nightly, keras-preprocessing, numpy, opt-einsum, protobuf, six, tensorboard, tensorflow-estimator, termcolor, typing-extensio
ns, wheel, wrapt
Required-by:
root@d9fa28761b2f:/app/waymo-adas-main/waymo-motion-prediction-2021#
root@d9fa28761b2f:/app/waymo-adas-main/waymo-motion-prediction-2021#
root@d9fa28761b2f:/app/waymo-adas-main/waymo-motion-prediction-2021# python3 -m pip show tensorflow-addons
Name: tensorflow-addons
Version: 0.10.0
Summary: TensorFlow Addons.
Home-page: UNKNOWN
Author: Google Inc.
Author-email: opensource@google.com
License: Apache 2.0
Location: /usr/local/lib/python3.6/dist-packages
Requires: typeguard
Required-by:
root@d9fa28761b2f:/app/waymo-adas-main/waymo-motion-prediction-2021# python3 -m pip show tensorflow-gpu
Name: tensorflow-gpu
Version: 2.0.0
Summary: TensorFlow is an open source machine learning framework for everyone.
Home-page: https://www.tensorflow.org/
Author: Google Inc.
Author-email: packages@tensorflow.org
License: Apache 2.0
Location: /usr/local/lib/python3.6/dist-packages
Requires: absl-py, astor, gast, google-pasta, grpcio, keras-applications, keras-preprocessing, numpy, opt-einsum, protobuf, six, tensorboard, tensorflow-estimator, termcolor, wheel, wrapt
Required-by:
root@d9fa28761b2f:/app/waymo-adas-main/waymo-motion-prediction-2021# python3 -m pip show tensorflow-estimator
Name: tensorflow-estimator
Version: 2.0.1
Summary: TensorFlow Estimator.
Home-page: https://www.tensorflow.org/
Author: Google Inc.
Author-email: UNKNOWN
License: Apache 2.0
Location: /usr/local/lib/python3.6/dist-packages
Requires:
Required-by: tensorflow, tensorflow-gpu
karanveersingh5623 commented 2 years ago

if you have any docker image , can you share ?

kbrodt commented 2 years ago

Did you try with another samples or redownload new uncompressed/tf_example/{training,validation,testing}?

No docker.

karanveersingh5623 commented 2 years ago

yea , I thought the same ….. so downloaded just 2 TF records again and tried …..same error . Reading TF records runs fine but when the time comes to write the records , there it fails...….. Don't know how can I proceed