search

NVIDIA / gpu-rest-engine

A REST API for Caffe using Docker and Go
BSD 3-Clause "New" or "Revised" License
421 stars 94 forks source link

34848 vs 0 error newer exported digits model with GRE #39

Closed rperdon closed 5 years ago

rperdon commented 5 years ago

Initializing Caffe classifiers F1018 03:33:43.747103 158 blob.cpp:496] Check failed: count_ == proto.data_size() (34848 vs. 0) Check failure stack trace: @ 0x7f9f6e3a75cd google::LogMessage::Fail() @ 0x7f9f6e3a9433 google::LogMessage::SendToLog() @ 0x7f9f6e3a715b google::LogMessage::Flush() @ 0x7f9f6e3a9e1e google::LogMessageFatal::~LogMessageFatal() @ 0x7f9f6e7901d2 caffe::Blob<>::FromProto() @ 0x7f9f6e76f091 caffe::Net<>::CopyTrainedLayersFrom() @ 0x7f9f6e777945 caffe::Net<>::CopyTrainedLayersFromBinaryProto() @ 0x7f9f6e777a27 caffe::Net<>::CopyTrainedLayersFrom() @ 0x619a5d Classifier::Classifier() @ 0x61acde classifier_initialize @ 0x615b27 _cgo_d8927303a3d1_C2func_classifier_initialize @ 0x45d650 (unknown) Aborted

I suspect this may be an issue with the older BVLC versus nvcaffe issue. My older models were trained in Digits 5.0, and my more recent ones now in 6.1.1 but noticed this error happening with the same data. Can you point me in a direction the classification.cpp file I can make changes to work with nvcaffe. I know this was a demo, but I'm working to develop this demo into a viable solution for our needs which the current TensorRT 5.0 is still not yet mature enough to fill. I have gone through the exercise of exporting a model to TensorRT, loading it on the inference server and found there is no way to "do inference" from a windows platform in TensorRT's current form. On top of this, models converted to TensorRT are not providing current results.

In examining the dockerfile, I see gre using bvlc caffe, which I'm wondering if a swap to nvcaffe can be done.

RUN git clone --depth 1 -b bvlc_inference https://github.com/flx42/caffe.git /caffe && \

I'm trying to find an nv inference example so I can hopefully replicate the structure done from this example.

rperdon commented 5 years ago

I figured out how to get Digits to output models in compatible format. In future, for anyone exploring this idea this error is caused by nvcaffe vs bvlc caffe. Digits 6.1.1 allows an option to change caffe formats.