apache / mxnet

Lightweight, Portable, Flexible Distributed/Mobile Deep Learning with Dynamic, Mutation-aware Dataflow Dep Scheduler; for Python, R, Julia, Scala, Go, Javascript and more
https://mxnet.apache.org
Apache License 2.0
20.77k stars 6.79k forks source link

mxnet cannot save parameter with name starting with slash #20405

Open khaotik opened 3 years ago

khaotik commented 3 years ago

Description

A minor issue: mx.numpy_extension.savez does not accept array-dict with a key starting with slash. This started to occur as soon as I upgraded to master fork. It is also inconsistent with numpy behavior.

This function is used by gluon.Block.save_parameters methods.

Error Message

Traceback (most recent call last):
  File "test.py", line 12, in <module>
    mx.numpy_extension.savez('/tmp/m_mx', **{param_name:m})
  File "/media/LNXDATA/WKSP/dev/incubator-mxnet/python/mxnet/numpy_extension/utils.py", line 122, in savez
    check_call(_LIB.MXNDArraySave(c_str(file), mx_uint(len(handles)), handles, keys))
  File "/media/LNXDATA/WKSP/dev/incubator-mxnet/python/mxnet/base.py", line 246, in check_call
    raise get_last_ffi_error()
mxnet.base.MXNetError: Traceback (most recent call last):
  File "/home/khaotik/WKSP/dev/incubator-mxnet/src/serialization/cnpy.cc", line 333
MXNetError: Check failed: mz_zip_writer_add_read_buf_callback(archive, blob_name_npy.data(), npy_header_blob_read_callback, static_cast<void*>(&callback_data), size_to_add, nullptr, nullptr, 0, MZ_NO_COMPRESSION, nullptr, 0, nullptr, 0): invalid filename

To Reproduce

#!/usr/bin/env python
import numpy as np
import mxnet as mx

m_npy = np.random.rand(256,256).astype('float32')
m = mx.ndarray.from_numpy(m_npy)
param_name = '/op/m' # starts with a slash

np.savez('/tmp/m_np', **{param_name:m_npy}) # WORKS
mx.numpy_extension.savez('/tmp/m_mx', **{param_name:m}) # BROKEN

What have you tried to solve it?

In file python/mxnet/numpy_extension/utils.py, substitute NDArraySave with NDArrayLegacySave provides a temp fix. Also, the issue can be avoided by not using slash as parameter name prefix.

Environment

ubuntu 20.04 anaconda py 3.8.3 mxnet master branch commit cb5bd4ea8b6fc9b568c13747bdb006ac047b72b5

leezu commented 3 years ago

npz format is based on the zip file standard. In zip files, filenames beginning with a / are not allowed.

4.4.17 file name: (Variable)

   4.4.17.1 The name of the file, with optional relative path.
   The path stored MUST NOT contain a drive or
   device letter, or a leading slash.  All slashes
   MUST be forward slashes '/' as opposed to
   backwards slashes '\' for compatibility with Amiga
   and UNIX file systems etc.  If input came from standard
   input, there is no file name field.  

https://pkware.cachefly.net/webdocs/casestudies/APPNOTE.TXT

For reference, we're using below zip implementation which, unlike Python / Numpy, enforces the filename requirements.

https://github.com/richgel999/miniz/blob/2281a42c77a08188a57609c8950b447ec62d6fb8/miniz_zip.c#L3109-L3113

Unfortunately this looks like a bug in numpy. We can consider following numpy here in writing non-standard zip files, but that's not supported by our current zip implementation.