search

MathMarEcol / pdyer_aus_bio

GNU General Public License v3.0
0 stars 0 forks source link

Memory issues for NbClust #32

Open PhDyellow opened 9 months ago

PhDyellow commented 9 months ago

Inital attempt: all indicies "alllong", at plotting resolution, 1/12

Result: Memory blowout, >64GB.

PhDyellow commented 9 months ago

only quick indices "all", plotting resolution 1/12

Result: Memory blowout >64Gb

PhDyellow commented 9 months ago

all indicies "alllong" at clustering resolution of 1/2

10 parallel branches only occupies about 2Gb RAM per worker.

Could probably push to 1/4 degree clustering resolution and manage.

PhDyellow commented 9 months ago

Some indices are very slow even at 1/2 degree resolution.

Currently GPlus has been running for 18 hours on a 1/2 degree Aus EEZ grid cell dataset, and it is only checking 2-5 clusters. I don't know how much longer it will take.