Errors while Training the model

[itcoderr]

Start Training...

         LoadedQueue is full (size =  15 ); stuff VideoReader back to WaitingQueue (size =  14 )
         LoadedQueue is full (size =  15 ); stuff VideoReader back to WaitingQueue (size =  4 )
         LoadedQueue is full (size =  15 ); stuff VideoReader back to WaitingQueue (size =  14 )

Training terminate at epoch: 0 , steps: 0 Queue info = listOfData.len() = 0; WaitingQueue.len() = 102; LoadedQueue.len() = 0; Pause = False

---

Empty Traceback (most recent call last) ~\Desktop\DATA\obj detection\ViolenceDetection-master\src\data\DataManager.py in AssignBatchData(self, batchData_) 270 batchData = self._queueForLoadedVideos.get(block=True, --> 271 timeout=dataSettings.TIMEOUT_FOR_WAIT_QUEUE) 272

~\Anaconda3\envs\tensorflow1\lib\queue.py in get(self, block, timeout) 171 if remaining <= 0.0: --> 172 raise Empty 173 self.not_empty.wait(remaining)

Empty:

During handling of the above exception, another exception occurred:

TimeoutError Traceback (most recent call last)

in () 193 if __name__ == "__main__": 194 main = Main() --> 195 main.Run() in Run(self) 62 63 while self.trainer.currentEpoch < trainSettings.MAX_TRAINING_EPOCH: ---> 64 self.trainer.PrepareNewBatchData() 65 self.trainer.Train(self.session) 66 self._trainCountInOneEpoch += 1 ~\Desktop\DATA\obj detection\ViolenceDetection-master\src\Trainer.py in PrepareNewBatchData(self) 75 print("\t Queue info = " + self._dataManager.GetQueueInfo()) 76 print() ---> 77 raise error 78 79 endGetBatchTime = time.time() ~\Desktop\DATA\obj detection\ViolenceDetection-master\src\Trainer.py in PrepareNewBatchData(self) 69 70 try: ---> 71 self._dataManager.AssignBatchData(self._batchData) 72 except TimeoutError as error: 73 print() ~\Desktop\DATA\obj detection\ViolenceDetection-master\src\data\DataManager.py in AssignBatchData(self, batchData_) 277 errorMessage += "\t TrainQueue info:\n" 278 errorMessage += self.GetQueueInfo() --> 279 raise TimeoutError(errorMessage) 280 281 batchData_.batchSize = batchData.batchSize TimeoutError: In TrainDataManager: Unable to get batch data in duration: 100(s) TrainQueue info: listOfData.len() = 0; WaitingQueue.len() = 102; LoadedQueue.len() = 0; Pause = False

[RimshaMajeed]

same error here aswell..

[JoshuaPiinRueyPan]

Hi, Please tell me the detail of your computer as well as the WAITING_QUEUE_MAX_SIZE and LOADED_QUEUE_MAX_SIZE in you settings/TrainSettings.py. The waitingQueue is a queue that contains to-do-list for the loading thread to load the video. While the loadedQueue is a queue that contains the loaded videos. In your case, it seems either you set the LOADED_QUEUE_MAX_SIZE too small or your graphic card is not powerful enough (so that the the network can't consume the loaded videos as fast as your loading thread). Also, it seems that you have 'data ping-pong' between the waitingQueue and the loadedQueue: when the video is loaded >> the network does not consume the videos on time >> the loadedQueue is full >> push the loaded queue to the waiting queue >> video is loaded >> loadedQueue is full >> ... To overcome this issue, maybe you should use fewer loading thread (NUMBER_OF_LOAD_DATA_THREADS in settings/TrainSettings.py) and adjust the WAITING_QUEUE_MAX_SIZE and LOADED_QUEUE_MAX_SIZE variables to fit your environment.

Note: The size of the WAITING_QUEUE_MAX_SIZE should be at least twice the size of the LOADED_QUEUE_MAX_SIZE.

[RimshaMajeed]

WAITING_QUEUE_MAX_SIZE = 180 LOADED_QUEUE_MAX_SIZE = 80 NUMBER_OF_LOAD_DATA_THREADS=4

Computer Details:

[JoshuaPiinRueyPan]

Hi @RimshaMajeed , How many CPU cores do you have? And also, do you have Nvidia Graphic card? Maybe the first thing to do is to set the NUMBER_OF_LOAD_DATA_THREADS=2 or even '1'. And see the outcome. Also, how many videos do you have?

[RimshaMajeed]

I have 22 Violent and NonViolent Videos each of 1 second. and,

[JoshuaPiinRueyPan]

Sorry, your hardware seems not be capable to perform the computation. For the CPU part, you can easily solve by setting the NUMBER_OF_LOAD_DATA_THREADS=1, and enlarging the TIMEOUT_FOR_WAIT_QUEUE (so that the process will not throw any timeout exception). However, without the Nvidia graphic card, the computational time may be quite long enough that I can't imagine...

If you are interested in the Computer Vision as well as the Deep Learning, it would be necessary to get a Nvidia graphic card (e.g. NV 1080Ti). Or, you can also take a look at Google Colab.

Also, looks like you have prepared your own dataset. However, with such amount of data, you can barely train a Deep Learning Mode. Fortunately, you can collect other violence videos from the internet, such as here and here. Train on such collected dataset first, then finetune on your own dataset (if you think your dataset should be learned especially). If you don't need the model to learn your videos especially, you can also combine the videos that you provided with other violence videos that can be found in the internet.

[onkareshwarprasad17]

@RimshaMajeed Were you able to solve your problem? I'm also having the same error

[onkareshwarprasad17]

@itcoderr Were you able to solve your problem? I'm also having the same error