TimoSaemann / caffe-segnet-cudnn5

This repository was a fork of BVLC/caffe and includes the upsample, bn, dense_image_data and softmax_with_loss (with class weighting) layers of caffe-segnet (https://github.com/alexgkendall/caffe-segnet) to run SegNet with cuDNN version 5.
Other
176 stars 127 forks source link

test_data_layer.bin fails #2

Open drahozin opened 7 years ago

drahozin commented 7 years ago

Issue summary

while running "make runtest" various errors are experienced The error looks like "floating"...

================= ERROR 1 ====================================== [----------] 4 tests from NetUpgradeTest [ RUN ] NetUpgradeTest.TestAllParams [ OK ] NetUpgradeTest.TestAllParams (1 ms) [ RUN ] NetUpgradeTest.TestUpgradeV1LayerType Error in `.build_release/test/test_all.testbin': munmap_chunk(): invalid pointer: 0x0000000000f00fe0 Aborted at 1483030739 (unix time) try "date -d @1483030739" if you are using GNU date PC: @ 0x7f70b5875267 (unknown) SIGABRT (@0x3e800000bb1) received by PID 2993 (TID 0x7f70b895ba40) from PID 2993; stack trace: @ 0x7f70b5c1ad10 (unknown) @ 0x7f70b5875267 (unknown) @ 0x7f70b5876eca (unknown) @ 0x7f70b58b8c53 (unknown) @ 0x7f70b58c49f8 (unknown) @ 0x7f70b64c426b caffe::BasePrefetchingDataLayer<>::~BasePrefetchingDataLayer() @ 0x7f70b652b562 boost::detail::sp_counted_impl_p<>::dispose() @ 0x65e589 caffe::NetUpgradeTest_TestUpgradeV1LayerType_Test::TestBody() @ 0x726993 testing::internal::HandleExceptionsInMethodIfSupported<>() @ 0x71eaca testing::Test::Run() @ 0x71ec18 testing::TestInfo::Run() @ 0x71ecf5 testing::TestCase::Run() @ 0x71f688 testing::internal::UnitTestImpl::RunAllTests() @ 0x71f953 testing::UnitTest::Run() @ 0x45aec2 main @ 0x7f70b5860a40 (unknown) @ 0x461fc9 _start @ 0x0 (unknown) Aborted (core dumped) Makefile:526: recipe for target 'runtest' failed

========================= ERROR 2 ================================ [----------] 12 tests from DataLayerTest/0, where TypeParam = caffe::CPUDevice [ RUN ] DataLayerTest/0.TestReshapeLevelDB Aborted at 1483030926 (unix time) try "date -d @1483030926" if you are using GNU date PC: @ 0x7fce07cae29b caffe::BasePrefetchingDataLayer<>::~BasePrefetchingDataLayer() SIGSEGV (@0x18) received by PID 3372 (TID 0x7fce0a145a40) from PID 24; stack trace: @ 0x7fce07404d10 (unknown) @ 0x7fce07cae29b caffe::BasePrefetchingDataLayer<>::~BasePrefetchingDataLayer() @ 0x5e86c8 caffe::DataLayerTest<>::TestReshape() @ 0x726993 testing::internal::HandleExceptionsInMethodIfSupported<>() @ 0x71eaca testing::Test::Run() @ 0x71ec18 testing::TestInfo::Run() @ 0x71ecf5 testing::TestCase::Run() @ 0x71f688 testing::internal::UnitTestImpl::RunAllTests() @ 0x71f953 testing::UnitTest::Run() @ 0x45aec2 main @ 0x7fce0704aa40 (unknown) @ 0x461fc9 _start @ 0x0 (unknown) Segmentation fault (core dumped) Makefile:526: recipe for target 'runtest' failed make: *** [runtest] Error 139

======================= ERROR 3 ======================================= [----------] 12 tests from DataLayerTest/0, where TypeParam = caffe::CPUDevice [ RUN ] DataLayerTest/0.TestReshapeLevelDB Aborted at 1483030926 (unix time) try "date -d @1483030926" if you are using GNU date PC: @ 0x7fce07cae29b caffe::BasePrefetchingDataLayer<>::~BasePrefetchingDataLayer() SIGSEGV (@0x18) received by PID 3372 (TID 0x7fce0a145a40) from PID 24; stack trace: @ 0x7fce07404d10 (unknown) @ 0x7fce07cae29b caffe::BasePrefetchingDataLayer<>::~BasePrefetchingDataLayer() @ 0x5e86c8 caffe::DataLayerTest<>::TestReshape() @ 0x726993 testing::internal::HandleExceptionsInMethodIfSupported<>() @ 0x71eaca testing::Test::Run() @ 0x71ec18 testing::TestInfo::Run() @ 0x71ecf5 testing::TestCase::Run() @ 0x71f688 testing::internal::UnitTestImpl::RunAllTests() @ 0x71f953 testing::UnitTest::Run() @ 0x45aec2 main @ 0x7fce0704aa40 (unknown) @ 0x461fc9 _start @ 0x0 (unknown) Segmentation fault (core dumped) Makefile:526: recipe for target 'runtest' failed make: *** [runtest] Error 139

========================= ERROR 4 ========================================== [----------] 12 tests from DataLayerTest/0, where TypeParam = caffe::CPUDevice [ RUN ] DataLayerTest/0.TestReadCropTrainSequenceUnseededLevelDB Aborted at 1483031372 (unix time) try "date -d @1483031372" if you are using GNU date PC: @ 0x7fbc78edd29b caffe::BasePrefetchingDataLayer<>::~BasePrefetchingDataLayer() SIGSEGV (@0x10) received by PID 5684 (TID 0x7fbc7b374a40) from PID 16; stack trace: @ 0x7fbc78633d10 (unknown) @ 0x7fbc78edd29b caffe::BasePrefetchingDataLayer<>::~BasePrefetchingDataLayer() @ 0x5ea549 caffe::DataLayerTest<>::TestReadCropTrainSequenceUnseeded() @ 0x726993 testing::internal::HandleExceptionsInMethodIfSupported<>() @ 0x71eaca testing::Test::Run() @ 0x71ec18 testing::TestInfo::Run() @ 0x71ecf5 testing::TestCase::Run() @ 0x71f688 testing::internal::UnitTestImpl::RunAllTests() @ 0x71f953 testing::UnitTest::Run() @ 0x45aec2 main @ 0x7fbc78279a40 (unknown) @ 0x461fc9 _start @ 0x0 (unknown) Segmentation fault (core dumped) Makefile:526: recipe for target 'runtest' failed make: *** [runtest] Error 139

TimoSaemann commented 7 years ago

I do not encounter errors while "make runtest". Using Ubuntu 14.04, Cuda 8.0, cuDNN v4, v5, v5.1, Titan X. How did you compile it? Make or CMake? Cuda version? cuDNN version? Which os do you use? Can you train and test SegNet anyway?

Vishruit commented 7 years ago

Hi, While running 'make runtest', I get the following error. Interesting, it compiles perfectly for the original version of caffe-segnet. I am using rise@karthik-biotech:~/Vishruit/soft/caffe-segnet$ uname -a Linux karthik-biotech 3.13.0-24-generic #47-Ubuntu SMP Fri May 2 23:30:00 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux and CUDA version = 7.5

On a separate note, if it helps: while make all and make test ran properly, initailly make runtest showed error for unable to find libcudart.so.7.5. I then updated the LD config path to include /usr/local/cuda/cuda7.5/lib64 to the LD path. But I didn't rerun the make all and make test after doing so.

Can you please help me in here? Thanks

It occurs in the DataLayerTest tests

[----------] 12 tests from DataLayerTest/3, where TypeParam = caffe::GPUDevice [ RUN ] DataLayerTest/3.TestReadCropTrainLevelDB Aborted at 1486477109 (unix time) try "date -d @1486477109" if you are using GNU date PC: @ 0x4756d0 boost::detail::sp_counted_base::release() SIGSEGV (@0x10) received by PID 8636 (TID 0x2b86b1cb4640) from PID 16; stack trace: @ 0x2b86b897e330 (unknown) @ 0x4756d0 boost::detail::sp_counted_base::release() @ 0x476bc3 caffe::Blob<>::~Blob() @ 0x2b86b7bceb51 caffe::BasePrefetchingDataLayer<>::~BasePrefetchingDataLayer() @ 0x4f05ca caffe::DataLayerTest<>::TestReadCrop() @ 0x8fcb33 testing::internal::HandleExceptionsInMethodIfSupported<>() @ 0x8f3817 testing::Test::Run() @ 0x8f38be testing::TestInfo::Run() @ 0x8f39c5 testing::TestCase::Run() @ 0x8f6d08 testing::internal::UnitTestImpl::RunAllTests() @ 0x8f6f97 testing::UnitTest::Run() @ 0x46c4bf main @ 0x2b86b8badf45 (unknown) @ 0x474119 (unknown) @ 0x0 (unknown) make: *** [runtest] Segmentation fault (core dumped)