search

Xilinx / Vitis-AI-Tutorials

MIT License
358 stars 144 forks source link

Caffe model trainning dog vs cat #60

Open maxmamaximus opened 2 years ago

maxmamaximus commented 2 years ago

Hello i start with tutorial on Vitis-AI and found some promblem on training state.

my environment is

Docker : vitis-ai-gpu:2.0.0.1103 GPU : RTX2080 CPU : Ryzen7 2700x OS : Ubuntu 18.04[WSL2]

Follow from 01-caffe_cats_vs_dogs --> In the 6 topic : Python and shell script , I'm testing script on this step but the .caffemodel isn't here So I train the model by myself from provided script but I stuck on training process when run this, like this image The log provided logfile_caffe_alexnetBNnoLRN.txt

From my continue debugging the tutorial i think it's stuck on the training process from caffe script link

Can you advise me for continue debug or solution for training AI from this state.

code-locker commented 2 years ago

Hi @maxmamaximus From your log I found that it is out of memory issue. This issue comes if you terminate some process in between and try to relaunch same process for the second or multiple times. Reason for this is, process fails to free() the used memory. Just restart your PC and try to perform the same. This may solve your issue. Thanks.

bhargavin1872008 commented 1 year ago

when running the requirements.txt of keras-yolov3-modelset -i 'm getting error for coremltools.it is showing like "couldn't find a version that satisfies the requirement tensorflow<=1.14 and tensorflow >=1.5(from tfcoremltools -r requirements.txt).(from version :2.2.0,2.2..1, 2.2.2, ...2.7.0rc0,2.7.0.rc1............) like this .can someone help me regarding this. Also ,i have a doubt .can we use ubuntu 20.04 ,cuda 11.7 ,cudnn 8.4.0 for this project. or have to use ubuntu 18.04,cuda 10.0 only which only works.please help me regarding this,i have less time in my hand.

maxmamaximus commented 1 year ago

Hi @code-locker Thank you from your suggestion. I'm trying to multiple time and reset my PC but it's not work for testing with RAM 64GB and VRAM 8 GB, When i'm change to my laptop(That's low spec than server). it's work but i'm confused.

maxmamaximus commented 1 year ago

Hi @bhargavin1872008 What versions you using Vitis-AI with your tutorial? You may down to proper environment with the tutorial.