Closed rpartsey closed 2 years ago
Hi @rpartsey!
It seems the problem is coming from pytorch which doesn't support uint16 and I wasn't aware of that. I think the first thing to do is to make sure pytorch developers know that there is a need. Knowing that a fix wouldn't come anytime soon I can suggest:
And yes you are completely write about the missing AllocationQuery in Convert. There is a TODO here that wasn't converted into an issue. (@andrewilyas do you have a bit of time to help out with this? I'm still trying to get everything sorted for v1.0)
Thanks for your report!
Thank you for your response, @GuillaumeLeclerc!
My comment was that ffcv allows to write data in uint16
using NDArrayField successfully, but there is no way to read it back (at least in the current release). Maybe it's worth adding a notification or warning (I mean not to use uint16
, better transform data to int16
because the range is the same).
Looks like that in torch community unsupported uint16
in the known issue and they are not going to change anything.
Yeah, we also thought about storing the data in int16.
I created #153 and will have it in v1. Feel free to reopen if you have more question
Hi dear maintainers and contributors,
My team and I (working on learned odometry estimation) found ffcv data loading speedup impressive and would like to integrate it into our training pipeline. In our original pipeline, we store depth images in
np.uint16
dtype, and we can successfully convert our dataset into.beton
format using theDatasetWriter
belowHowever, we have trouble reading the depth from
.beton
at data loading time astorch
doesn't supportuint16
. If we defineLoader
aswe get
We also tried defining
Loader
aswe get
We would be grateful if you could answer the questions regarding the issue described above:
uint16
dtype?AllocationQuery
We appreciate any help you can provide. Looking forward to reading your paper.