Significantly improve subsetting performance by using GNU parallel.
Subset used on srs_sst_l3s_1d_day_gridded_url (345 files in total): TIME,2014-02-21T03:20:00.000Z,2015-02-01T03:20:00.000999Z;LATITUDE,-23.32,-23.05;LONGITUDE,113.64,113.86
Single threaded:
Start: 16:53:29
End: 16:59:58
Total: 6m29s
Multi threaded (GNU parallel):
Start: 17:15:42
End: 17:16:29
Total: 47s
That's roughly a x10 speed-up.
And for a relatively big aggregation @kereid was experimenting with:
TIME,2010-02-21T03:20:00.000Z,2015-02-01T03:20:00.000999Z;LATITUDE,-23.32,-23.05;LONGITUDE,113.64,113.86
Single threaded, 1800 files in total:
Start: 11:30:45
End: 12:07:52
Total: 37m7s
Multi threaded (GNU parallel), 1800 files in total:
Start: 17:32:14
End: 17:50:18
Total: 18m4s
That's roughly a x2 speed-up. Looks like for big aggregations we're still I/O bound.
dnahodil
commented
9 years ago
Significantly improve subsetting performance by using GNU parallel.
Subset used on
srs_sst_l3s_1d_day_gridded_url(345 files in total): TIME,2014-02-21T03:20:00.000Z,2015-02-01T03:20:00.000999Z;LATITUDE,-23.32,-23.05;LONGITUDE,113.64,113.86Single threaded: Start: 16:53:29 End: 16:59:58 Total: 6m29s
Multi threaded (GNU parallel): Start: 17:15:42 End: 17:16:29 Total: 47s
That's roughly a x10 speed-up.
And for a relatively big aggregation @kereid was experimenting with: TIME,2010-02-21T03:20:00.000Z,2015-02-01T03:20:00.000999Z;LATITUDE,-23.32,-23.05;LONGITUDE,113.64,113.86
Single threaded, 1800 files in total: Start: 11:30:45 End: 12:07:52 Total: 37m7s
Multi threaded (GNU parallel), 1800 files in total: Start: 17:32:14 End: 17:50:18 Total: 18m4s
That's roughly a x2 speed-up. Looks like for big aggregations we're still I/O bound.