Train performances improvement

ocourtin commented 6 years ago

While training GPU spend lot of time idle, mainly for two reasons:

Lot of time spend with CPU computation (scikit metrics)
No enough workers to feed data

This PR fix both.

And as a result, using several GPU for training become meaningful.

ocourtin commented 6 years ago

Indeed no longer need scikit stuff.

About predict, it's a bit better, but guess could be improve again (GPU-Utilisation and multi GPU). Will give a look, but not on my top list for now.

Launch black, without option, on metrics.py file (if i do so on train.py it would format code i did'nt touch...) Dunno if you use black with specific options

ocourtin commented 6 years ago

My environnement is not used only for robosat, so pip freeze return quite others pip modules already installed. So not very helpful.

But if i look only on deps who are explicitly imported later in code, it could be reduce as:

attrs==18.1.0 Flask==1.0.2 geojson==2.4.0 matplotlib==2.2.2 mercantile==1.0.4 numpy==1.14.4 osmium==2.14.1 py==1.5.3 pyproj==1.9.5.1 rasterio==1.0b1 requests==2.18.4 Rtree==0.8.3 Shapely==1.6.4.post1 supermercado==0.0.5 toml==0.9.4 torch==0.4.0 torchvision==0.2.1 tqdm==4.23.4

Thanks for the black options setting. 9d9baa4 commit was already OK for both metrics.py and train.py

ocourtin commented 6 years ago

Hum meant bc27447

daniel-j-h commented 6 years ago

Thanks again, this is awesome! :tada:

mapbox / robosat

Train performances improvement #112