search

ShuangLI59 / person_search

Joint Detection and Identification Feature Learning for Person Search
https://arxiv.org/abs/1604.01850
Other
738 stars 241 forks source link

Error pretrain #3

Closed pribadihcr closed 7 years ago

pribadihcr commented 8 years ago

Hi @ShuangLI59

When I tried to run experiments/scripts/pretrain.sh, I got the following error: I0723 15:55:34.441007 21095 layer_factory.hpp:77] Creating layer data I0723 15:55:34.441382 21095 net.cpp:106] Creating Layer data I0723 15:55:34.441409 21095 net.cpp:411] data -> data I0723 15:55:34.441429 21095 net.cpp:411] data -> label I0723 15:55:34.442243 21105 db_lmdb.cpp:38] Opened lmdb data/psdb/pretrain_db/train_lmdb terminate called after throwing an instance of 'std::length_error' what(): basic_string::_S_create * Aborted at 1469260534 (unix time) try "date -d @1469260534" if you are using GNU date * PC: @ 0x7f6e23885c37 (unknown) * SIGABRT (@0x3e800005267) received by PID 21095 (TID 0x7f6defdcf700) from PID 21095; stack trace: * @ 0x7f6e23885cb0 (unknown) @ 0x7f6e23885c37 (unknown) @ 0x7f6e23889028 (unknown) @ 0x7f6e2463b535 (unknown) @ 0x7f6e246396d6 (unknown) @ 0x7f6e24639703 (unknown) @ 0x7f6e24639922 (unknown) @ 0x7f6e2468b3a7 (unknown) @ 0x7f6e24695262 (unknown) @ 0x7f6e24696971 (unknown) @ 0x7f6e24696a2d (unknown) @ 0x7f6e2546a89a caffe::db::LMDBCursor::value() @ 0x7f6e2533e5fb caffe::DataReader::Body::read_one() @ 0x7f6e2533e9c4 caffe::DataReader::Body::InternalThreadEntry() @ 0x7f6e25484180 caffe::InternalThread::entry() @ 0x7f6e1cbbea4a (unknown) @ 0x7f6e178cb184 start_thread @ 0x7f6e2394937d (unknown) @ 0x0 (unknown)

Many thanks for help

Cysu commented 8 years ago

It seems that the lmdb is corrupted. What are the sizes of the lmdbs (by du -sh data/psdb/pretrain_db/*_lmdb)? Are there any errors in executing the data generation script? scripts/make_db.sh /path/to/the/downloaded/dataset.zip

nhquanst commented 8 years ago

Hi @ShuangLI59 I am using GPU K1100M and VGG16_iter_50000.caffemodel

When i trained with "experiments/scripts/train.sh 0" command. I got following error:

Solving... F0808 16:14:54.009300 23672 syncedmem.cpp:56] Check failed: error == cudaSuccess (2 vs. 0) out of memory * Check failure stack trace: * experiments/scripts/train.sh: line 46: 23672 Aborted (core dumped) python2 tools/train_net.py --gpu ${GPU_ID} --solver models/${PT_DIR}/${NET}/solver.prototxt --weights output/${DATASET}_pretrain/${NET}_iter_50000.caffemodel --imdb ${TRAIN_IMDB} --iters ${ITERS} --cfg experiments/cfgs/train.yml ${EXTRA_ARGS}

I think my GPU memory not enough. May i use another pretrain model?

Cysu commented 8 years ago

Hi, @nhquanst. Sorry but we currently only use VGG-16 as the pretrained model, which requires at least 6GB memory I think.

nhquanst commented 8 years ago

@Cysu, thanks a lot! I will try with another GPU.

duanLH commented 8 years ago

@nhquanst @pribadihcr @ShuangLI59 Do you run the code successfully now?

nhquanst commented 8 years ago

sorry, not yet. Because of limitation of my server hardware

On Wed, Aug 17, 2016 at 5:06 PM, duanLH notifications@github.com wrote:

@nhquanst https://github.com/nhquanst @pribadihcr https://github.com/pribadihcr @ShuangLI59 https://github.com/ShuangLI59 Do you run the code successfully now?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/ShuangLI59/person_search/issues/3#issuecomment-240369409, or mute the thread https://github.com/notifications/unsubscribe-auth/APiUHJskH2UH6rKoNiKr1s_sCLihy8YMks5qgt0UgaJpZM4JTT4m .

Cysu commented 7 years ago

We have improved our method significantly and now we use a better version of caffe that provides memory optimization. Maybe you could give it a try again. And please feel free to reopen this issue if there are further problems.

nhquanst commented 7 years ago

thank you very much, i will try it as soon as possible

On Tue, Feb 28, 2017 at 6:32 PM, Tong Xiao notifications@github.com wrote:

Closed #3 https://github.com/ShuangLI59/person_search/issues/3.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/ShuangLI59/person_search/issues/3#event-979990862, or mute the thread https://github.com/notifications/unsubscribe-auth/APiUHGa5Rj1sv6WbZXEOMhsg8UukqY5xks5rhAXngaJpZM4JTT4m .