search

CEA-LIST / N2D2

N2D2 is an open source CAD framework for Deep Neural Network simulation and full DNN-based applications building.
Other
146 stars 36 forks source link

Issue while running MNIST #118

Closed dipsankarb closed 1 year ago

dipsankarb commented 1 year ago

Hi team,

When I am trying to run n2d2 on MNIST, i am getting this error:

./../build/bin/n2d2 "mnist24_16c4s2_24c5s2_150_10.ini" Loading network configuration file mnist24_16c4s2_24c5s2_150_10.ini Notice: Unused section sp in INI file Notice: Unused section sp.Transformation in INI file Notice: Unused section conv1 in INI file Notice: Unused section conv2 in INI file Notice: Unused section fc1 in INI file Notice: Unused section fc1.drop in INI file Notice: Unused section fc2 in INI file Notice: Unused section softmax in INI file Notice: Unused section softmax.Target in INI file Notice: Unused section common.config in INI file Time elapsed: 0.0112938 s Error: Wrong file format for images file: /raid/home/dip/N2D2/dataset/mnist/train-images-idx3-ubyte

Can you please help me understand where i many be going wrong:

n2d2 -v output :

Option -v: display version information N2D2 (Sep 19 2022 17:00:35) (C) Copyright 2010-2019 CEA LIST. All Rights Reserved.

GNU GCC/G++ compiler version: 9.3.0

General configuration for OpenCV 4.5.5 ===================================== Version control: 4.5.5

Platform: Timestamp: 2022-09-19T06:02:52Z Host: Linux 5.4.0-52-generic x86_64 CMake: 3.16.3 CMake generator: Unix Makefiles CMake build tool: /usr/bin/make Configuration: Release

CPU/HW features: Baseline: SSE SSE2 SSE3 requested: SSE3 Dispatched code generation: SSE4_1 SSE4_2 FP16 AVX AVX2 AVX512_SKX requested: SSE4_1 SSE4_2 AVX FP16 AVX2 AVX512_SKX SSE4_1 (18 files): + SSSE3 SSE4_1 SSE4_2 (2 files): + SSSE3 SSE4_1 POPCNT SSE4_2 FP16 (1 files): + SSSE3 SSE4_1 POPCNT SSE4_2 FP16 AVX AVX (5 files): + SSSE3 SSE4_1 POPCNT SSE4_2 AVX AVX2 (33 files): + SSSE3 SSE4_1 POPCNT SSE4_2 FP16 FMA3 AVX AVX2 AVX512_SKX (8 files): + SSSE3 SSE4_1 POPCNT SSE4_2 FP16 FMA3 AVX AVX2 AVX_512F AVX512_COMMON AVX512_SKX

C/C++: Built as dynamic libs?: YES C++ standard: 11 C++ Compiler: /usr/bin/c++ (ver 9.3.0) C++ flags (Release): -fsigned-char -W -Wall -Werror=return-type -Werror=non-virtual-dtor -Werror=address -Werror=sequence-point -Wformat -Werror=format-security -Wmissing-declarations -Wundef -Winit-self -Wpointer-arith -Wshadow -Wsign-promo -Wuninitialized -Wsuggest-override -Wno-delete-non-virtual-dtor -Wno-comment -Wimplicit-fallthrough=3 -Wno-strict-overflow -fdiagnostics-show-option -Wno-long-long -pthread -fomit-frame-pointer -ffunction-sections -fdata-sections -msse -msse2 -msse3 -fvisibility=hidden -fvisibility-inlines-hidden -O3 -DNDEBUG -DNDEBUG C++ flags (Debug): -fsigned-char -W -Wall -Werror=return-type -Werror=non-virtual-dtor -Werror=address -Werror=sequence-point -Wformat -Werror=format-security -Wmissing-declarations -Wundef -Winit-self -Wpointer-arith -Wshadow -Wsign-promo -Wuninitialized -Wsuggest-override -Wno-delete-non-virtual-dtor -Wno-comment -Wimplicit-fallthrough=3 -Wno-strict-overflow -fdiagnostics-show-option -Wno-long-long -pthread -fomit-frame-pointer -ffunction-sections -fdata-sections -msse -msse2 -msse3 -fvisibility=hidden -fvisibility-inlines-hidden -g -O0 -DDEBUG -D_DEBUG C Compiler: /usr/bin/cc C flags (Release): -fsigned-char -W -Wall -Werror=return-type -Werror=address -Werror=sequence-point -Wformat -Werror=format-security -Wmissing-declarations -Wmissing-prototypes -Wstrict-prototypes -Wundef -Winit-self -Wpointer-arith -Wshadow -Wuninitialized -Wno-comment -Wimplicit-fallthrough=3 -Wno-strict-overflow -fdiagnostics-show-option -Wno-long-long -pthread -fomit-frame-pointer -ffunction-sections -fdata-sections -msse -msse2 -msse3 -fvisibility=hidden -O3 -DNDEBUG -DNDEBUG C flags (Debug): -fsigned-char -W -Wall -Werror=return-type -Werror=address -Werror=sequence-point -Wformat -Werror=format-security -Wmissing-declarations -Wmissing-prototypes -Wstrict-prototypes -Wundef -Winit-self -Wpointer-arith -Wshadow -Wuninitialized -Wno-comment -Wimplicit-fallthrough=3 -Wno-strict-overflow -fdiagnostics-show-option -Wno-long-long -pthread -fomit-frame-pointer -ffunction-sections -fdata-sections -msse -msse2 -msse3 -fvisibility=hidden -g -O0 -DDEBUG -D_DEBUG Linker flags (Release): -Wl,--exclude-libs,libippicv.a -Wl,--exclude-libs,libippiw.a -Wl,--gc-sections -Wl,--as-needed
Linker flags (Debug): -Wl,--exclude-libs,libippicv.a -Wl,--exclude-libs,libippiw.a -Wl,--gc-sections -Wl,--as-needed
ccache: NO Precompiled headers: NO Extra dependencies: dl m pthread rt 3rdparty dependencies:

OpenCV modules: To be built: calib3d core dnn features2d flann gapi highgui imgcodecs imgproc ml objdetect photo stitching ts video videoio Disabled: world Disabled by dependency: - Unavailable: java python2 python3 Applications: tests perf_tests apps Documentation: NO Non-free algorithms: NO

GUI: NONE GTK+: NO VTK support: NO

Media I/O: ZLib: /usr/lib/x86_64-linux-gnu/libz.so (ver 1.2.11) JPEG: libjpeg-turbo (ver 2.1.2-62) WEBP: build (ver encoder: 0x020f) PNG: build (ver 1.6.37) TIFF: build (ver 42 - 4.2.0) JPEG 2000: build (ver 2.4.0) OpenEXR: build (ver 2.3.0) HDR: YES SUNRASTER: YES PXM: YES PFM: YES

Video I/O: DC1394: NO FFMPEG: NO avcodec: NO avformat: NO avutil: NO swscale: NO avresample: NO GStreamer: NO v4l/v4l2: YES (linux/videodev2.h)

Parallel framework: pthreads

Trace: YES (with Intel ITT)

Other third-party libraries: Intel IPP: 2020.0.0 Gold [2020.0.0] at: /raid/home/dip/opencv/build/3rdparty/ippicv/ippicv_lnx/icv Intel IPP IW: sources (2020.0.0) at: /raid/home/dip/opencv/build/3rdparty/ippicv/ippicv_lnx/iw VA: NO Lapack: NO Eigen: NO Custom HAL: NO Protobuf: build (3.19.1)

OpenCL: YES (no extra features) Include path: /raid/home/dip/opencv/3rdparty/include/opencl/1.2 Link libraries: Dynamic load

Python (for build): /usr/bin/python2.7

Java:
ant: NO JNI: NO Java wrappers: NO Java tests: NO

Install to: /raid/home/dip/opencv/install

Detected 8 CUDA Capable device(s)

Device #0: "A100 Graphics Device" CUDA Driver Version / Runtime Version: 11.0 / 11.2 CUDA Capability Major/Minor version number: 8.0

CuDNN version: 8.4.1

vtemplier commented 1 year ago

Hi @dipsankarb,

Indeed, you're right when one uses the installation dataset python script tools/install/install_dataset.py to download MNIST, it is impossible to read the dataset then.

I found that it is linked to the manner the extraction has been made. According to this forum, using shutil.copyfileobj(f_in, f_out) gives a magic number of 529205256 while using gzip gives a magic number of 2051 as expected by the dataset

TRAINING SET IMAGE FILE (train-images-idx3-ubyte):
[offset] [type]          [value]          [description]
0000     32 bit integer  0x00000803(2051) magic number
0004     32 bit integer  60000            number of images
0008     32 bit integer  28               number of rows
0012     32 bit integer  28               number of columns
0016     unsigned byte   ??               pixel
0017     unsigned byte   ??               pixel
........
xxxx     unsigned byte   ??               pixel

So please update your tools/install/install_dataset.py script with the following lines and re-download the MNIST dataset.

    elif filename.endswith(GZ_EXTENSION):
        print("Extracting gz file.")
        raw = gzip.open(file_location, 'rb').read()
        open(os.path.splitext(file_location)[0], 'wb').write(raw)

I will also fix it in the repository.

Thank you for your feedback and let me know if you managed to use MNIST.

dipsankarb commented 1 year ago

Yes this worked. Thanks for the help, have to import gzip library also.

Another error is now cropping up related to gnuplot

gnuplot> if (!exists("multiplot")) set term png size 800,600 enhanced large ^ line 0: unknown or ambiguous terminal type; type just 'set terminal' for a list

WARNING: Plotting with an 'unknown' terminal. No output will be generated. Please select a terminal with 'set terminal'. WARNING: Plotting with an 'unknown' terminal. No output will be generated. Please select a terminal with 'set terminal'. CUBLAS initialized on device #0 Learning #2400 (5%) 50.87% at 2044.81 p./s ( 122689 p./min)

vtemplier commented 1 year ago

Indeed, it seems you have a problem with Gnuplot in your environment.

Perhaps the documentation of Gnuplot has a solution for your problem

4.4 Arranging multiple plots next to each other on a single page
The command you want is set multiplot. The program can place a specified number of plots on a regular grid (set mulplot layout <rows>, <columns> ...) or you can position them one by one using set origin and set size.

Otherwise here some forums that, I hope, will help you

dipsankarb commented 1 year ago

Issue is solved now. I had to enable png for gnuplot in addition to setting the loadpath. Thanks for your help.