ExpHP
commented
5 years ago Summary of all changes
Additions / Big changes
The new serialization traits
Serializable has been split up quite a bit, because not all operations can be done on all types.
/// Trait that permits reading a type from an `.npy` file.
pub trait Deserialize: Sized {
/// Think of this as a `Fn(&[u8]) -> Self`, with bonuses.
///
/// Unfortunately, until rust supports existential associated types, actual closures
/// cannot be used here, and you must define something that manually implements [`TypeRead`].
type Reader: TypeRead<Value=Self>;
/// Get a function that deserializes a single data field at a time
///
/// The function receives a byte buffer containing at least
/// `dtype.num_bytes()` bytes.
///
/// # Errors
///
/// Returns `Err` if the `DType` is not compatible with `Self`.
fn reader(dtype: &DType) -> Result<Self::Reader, DTypeError>;
}
/// Trait that permits writing a type to an `.npy` file.
pub trait Serialize {
/// Think of this as some sort of `for<W: io::Write> Fn(W, &Self) -> io::Result<()>`.
///
/// Unfortunately, rust does not have generic closures, so you must manually define
/// your own implementor of the [`TypeWrite`] trait.
type Writer: TypeWrite<Value=Self>;
/// Get a function that serializes a single data field at a time.
///
/// # Errors
///
/// Returns `Err` if the `DType` is not compatible with `Self`.
fn writer(dtype: &DType) -> Result<Self::Writer, DTypeError>;
}
/// Subtrait of [`Serialize`] for types which have a reasonable default `DType`.
///
/// This opens up some simpler APIs for serialization. (e.g. [`::to_file`])
pub trait AutoSerialize: Serialize {
/// A suggested format for serialization.
///
/// The builtin implementations for primitive types generally prefer `|` endianness if possible,
/// else the machine endian format.
fn default_dtype() -> DType;
}Deserializeis implemented for primitive ints, floats, andVec<u8>. (but not[u8], which is unsized)Serializeis implemented for primitive ints, floats,Vec<u8>,[u8], and for things behind a variety of pointer types.AutoSerializeis implemented for primitive ints and floats. (but notVec<u8>/[u8], for which|Snand|Vnboth sound reasonable and both have disadvantages)
Where's n_bytes?
It's on DType now. There is no other way to possibly support e.g. |V42.
impl DType {
pub fn num_bytes(&self) -> usize;
}Worth noting is that, unfortunately, this means the compiler can no longer constant-fold the sizes for large records. To mitigate that, we now have...
Two-stage de/serialization
So, what's the deal with the Reader and Writer types? Basically, types now need to validate their DTypes and possibly do different things based on what it contains.
To ensure that this can be done efficiently, de/serialization now takes place in two stages:
- DType validation. (and potentially caching useful info like offsets)
- The actual reading/writing.
The first step is, of course, done by Serialize::writer and Deserialize::reader already seen above. The second step is done by these methods:
pub trait TypeRead {
type Value;
fn read_one<'a>(&self, bytes: &'a [u8]) -> (Self::Value, &'a [u8]);
}
pub trait TypeWrite {
type Value: ?Sized;
fn write_one<W: io::Write>(&self, writer: W, value: &Self::Value) -> io::Result<()>
where Self: Sized;
}Needless to say, manually implementing these traits has become a fair bit of a chore now. Please see the updated roundtrip example.
The error type exposes a single public constructor for use by manual impls of the traits.
/// Indicates that a particular rust type does not support serialization or deserialization
/// as a given [`DType`].
#[derive(Debug, Clone)]
pub struct DTypeError(ErrorKind);
impl fmt::Display for DTypeError { ... }
impl std::error::Error for DTypeError { ... }
impl DTypeError {
pub fn custom<S: AsRef<str>>(msg: S) -> Self;
}One more trait: TypeWriteDyn
There's... this... thing.
pub trait TypeWriteDyn: TypeWrite {
#[doc(hidden)]
fn write_one_dyn(&self, writer: &mut dyn io::Write, value: &Self::Value) -> io::Result<()>;
}
impl<T: TypeWrite> TypeWriteDyn for T { }Long story short: dyn TypeWrite can't do anything, so you should use dyn TypeWriteDyn instead.
I'd remove this from the PR if I could to save it for later consideration... but currently some of the built-in impls use it. I need to do some benchmarking first.
Outfile::open_with_dtype
The existing methods for writing files require AutoSerialize instead of Serialize. One new method, OutFile::open_with_dtype, was added for types that only implement the latter.
impl<Row: AutoSerialize> OutFile<Row> {
/// Create a file, using the default format for the given type.
pub fn open<P: AsRef<Path>>(path: P) -> io::Result<Self>;
}
impl<Row: Serialize> OutFile<Row> {
pub fn open_with_dtype<P: AsRef<Path>>(dtype: &DType, path: P) -> io::Result<Self>;
}
pub fn to_file<S, T, P>(filename: P, data: T) -> ::std::io::Result<()> where
P: AsRef<Path>,
S: AutoSerialize,
T: IntoIterator<Item=S>;DType::Plain.ty changed from String -> TypeStr
The new TypeStr type is a fully parsed form of a stringlike descr. This is necessary so that reader and writer can easily match on various properties of the descr without having to look at string text.
/// Represents an Array Interface type-string.
///
/// This is more or less the `DType` of a scalar type.
#[derive(Debug, Clone, PartialEq, Eq, PartialOrd, Ord, Hash)]
pub struct TypeStr {
/* no public fields */
}
impl fmt::Display for TypeStr { ... }
impl str::FromStr for TypeStr {
type Error = ParseTypeStrError;
...
}Parsing one generally produces validation errors for anything not accepted by the np.dtype function.
It currently exposes no public API for inspection or manipulation.
The error is just a garden-variety error type (Debug, Clone, Display, FromStr). I purposefully didn't use nom because nom gives terrible error messages.
There is a helper for "upgrading" a TypeStr to a DType:
impl DType {
/// Construct a scalar `DType`. (one which is not a nested array or record type)
pub fn new_scalar(ty: TypeStr) -> Self;
}"derive" feature
This is a replacement for #[macro_use] extern crate npy_derive. Basically, we don't want #[derive(Serialize)] to clash with serde or other crates, so we instead recommend the following setup to users:
[dependencies]
npy-rs = { version = "0.5", features = ["derive"] }extern crate npy;
#[derive(npy::Serialize, npy::Deserialize)]
struct MyStruct { a: i32, b: f64 }Notice the above works even for 2015 edition crates. (the examples and doctests should attest to this!)
NpyData::dtype
impl<'a, T: Deserialize> NpyData<'a, T> {
pub fn dtype(&self) -> DType;
}This is necessary in order to be able to read an NPY file and then write a new one back that has the same format. I also wanted it for some of the tests.
Little Things
- travis now checks that the
examples/build (because I had to update.travis.ymlto add the feature flag, and at that point... why not?) DTypenow derivesClonebecause I needed it in the derive macro.npy_derivenow depends onproc_macro2, whose types appear in the public API ofsynandquote- unnecessary lifetime deleted from
to_file
...and I think that's everything. (whew!)
Edit 1: Updated the signatures of TypeRead to reflect the new, faster API
Edit 2:
- Removed
TypeRead::read_one_into(it can be added backwards compatibly later) - Removed "Helpers for creating
|Snand|Vn" (it can be added backwards compatibly later) - Added
NpyData::dtype
xd009642
This is now ready for review!
I have chosen to finally submit a PR now that
cargo test --allsucceeds, and the behavior of all of the existing examples has been verified. Some work still remains to be done.Overview
|Snand binary blobs|Vn(asVec<u8>)<M8[us]and timedeltas<m8[us](asu64andi64)Serializablehad to be split into three traits to accomodate these features. More about this later.np.float16,np.float128, and unicode strings in the future.This is a large PR. However, as of Thursday June 6, I rewrote the commit history so that each commit can be easily reviewed. (there is no longer anything introduced in one commit that gets changed in another)
Closes #11. Closes #12. Closes #19.
I will add a comment which itemizes all of the changes to the public API and the reasoning behind them.