search

pjreddie / darknet

Convolutional Neural Networks
http://pjreddie.com/darknet/
Other
25.81k stars 21.33k forks source link

Enhancement idea: SQL lite mgmt of training images #107

Open rtrahms opened 7 years ago

rtrahms commented 7 years ago

Hi - Darknet/YOLOv2 is great!! We've trained it on a number of datasets, with really good results.

When managing a large number of images, it would be great to have an option to move beyond pathname indicators for referencing files (in train.txt and test.txt).

Any thoughts on using a light database approach to managing files (like NoSQL or SQL-lite)? Could make managing even larger datasets easier.

Thanks again for an awesome framework! Rob

goatShaver commented 7 years ago

I have considered, specifically because I have a lot of well 'measured' data such as angles (from many dimensions), lighting situations (I have a light meter) etc. We are still in research/academic stage and I'm finding that people new to the system like text files, or json files etc. because they can see the format and easily process it. There still exist a large number of people uncomfortable with databases and the overhead associated with learning or maintaining.

The larger issue for SQLite is that you have a file which must be 'managed' so that everyone can read and if you update it, everyone must have the updated version. This turns out to be a logistics nightmare. As much as I love SQLite, for team projects, it is tough - you really need a SQL server (ideally a cloud based) server.

I would be interested in what type of thing you'd put in the database and how you see it working to know if there are advantages that outweigh the disadvantages. (I'm a fan of your idea - just haven't figured out the model where it makes the most sense.)