search

patlevin / tfjs-to-tf

A TensorFlow.js Graph Model Converter
MIT License
136 stars 19 forks source link

ValueError: buffer is smaller than requested size #2

Closed withtypes closed 4 years ago

withtypes commented 4 years ago

I'm trying to convert this bodypix model (resnet50, stride 32). It was already mentioned in #1 but this seems to be a different error:

Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/tfjs_graph_converter/converter.py", line 137, in main
    convert(argv[0].split(' '))
  File "/usr/local/lib/python3.6/dist-packages/tfjs_graph_converter/converter.py", line 107, in convert
    api.graph_model_to_frozen_graph(args.input_path, args.output_path)
  File "/usr/local/lib/python3.6/dist-packages/tfjs_graph_converter/api.py", line 217, in graph_model_to_frozen_graph
    graph = load_graph_model(model_dir)
  File "/usr/local/lib/python3.6/dist-packages/tfjs_graph_converter/api.py", line 204, in load_graph_model
    return _convert_graph_model_to_graph(model_json, model_dir)
  File "/usr/local/lib/python3.6/dist-packages/tfjs_graph_converter/api.py", line 177, in _convert_graph_model_to_graph
    weight_list = read_weights(weights_manifest, base_path, flatten=True)
  File "/usr/local/lib/python3.6/dist-packages/tensorflowjs/read_weights.py", line 74, in read_weights
    return decode_weights(weights_manifest, data_buffers, flatten=flatten)
  File "/usr/local/lib/python3.6/dist-packages/tensorflowjs/read_weights.py", line 189, in decode_weights
    value = _deserialize_numeric_array(data_buffer, offset, dtype, shape)
  File "/usr/local/lib/python3.6/dist-packages/tensorflowjs/read_weights.py", line 123, in _deserialize_numeric_array
    offset=offset).reshape(shape)
ValueError: buffer is smaller than requested size

I've downloaded the weights with

for i in {1..23}; do wget https://storage.googleapis.com/tfjs-models/savedmodel/bodypix/resnet50/float/group1-shard${i}of23.bin; done

Versions used are

tfjs_graph_converter 0.2.0
Dependency versions:
    tensorflow 1.15.0
    tensorflowjs 1.3.2
patlevin commented 4 years ago

Please check the sizes of the downloaded weight files. You'll get this error if the weight files are invalid. The files should have a size of exactly 4MB (4096kB), except for the last file (group1-shard23of23,bin), which should be a little smaller.

The problem stems from wget not decompressing the downloaded files (the server delivers the files gzip-compressed). I don't know the correct switch for wget, but this curl command should do the trick:

curl -O --compressed https://storage.googleapis.com/tfjs-models/savedmodel/bodypix/resnet50/float/group1-shard[1-23]of23.bin
withtypes commented 4 years ago

Ah, yes, you are correct! wget messed up. It works now. Thank you