Memory error&invalid value encountered in true_divide in K-means

[merkato]

Once again it's me ;) QGIS 2.99/SCP 6.0.1

I have some issues with K-Means Clustering and Isodata probably:

I'm using Sentinel-2 data as cropped TIFF (about 1/4 of granule).

"Błąd [28] : Memory error. Please, set a lower value of RAM in the tab Settings"

I've set to 2000, and 1000 accordingly. Checking system usage, there's no more than 3 GB RAM used in that time on 8 GB system.

QGIS throws out these messages:

Python warnings tab:

2018-01-30T20:23:11 WARNING warning:/home/mechanik/.local/share/QGIS/QGIS3/profiles/default/python/plugins/SemiAutomaticClassificationPlugin/core/utils.py:962: FutureWarning:

            comparison to `None` will result in an elementwise object comparison in the future.

            traceback:  File "/home/mechanik/.local/share/QGIS/QGIS3/profiles/default/python/plugins/SemiAutomaticClassificationPlugin/maininterface/clusteringTab.py", line 120, in calculateClusteringAction          self.calculateClustering()
              File "/home/mechanik/.local/share/QGIS/QGIS3/profiles/default/python/plugins/SemiAutomaticClassificationPlugin/maininterface/clusteringTab.py", line 193, in calculateClustering
                self.kmeansCalculation(rD, bL, clssOut, k_or_sigs, iterations, thresh, NoDataValue, batch, bandSetNumber)
              File "/home/mechanik/.local/share/QGIS/QGIS3/profiles/default/python/plugins/SemiAutomaticClassificationPlugin/maininterface/clusteringTab.py", line 671, in kmeansCalculation
                r, sigs, sL, distances = self.kmeansIteration(rD, bL, iteration, k_or_sigs, iterations, thresh, NoDataValue, batch, bandSetNumber)
              File "/home/mechanik/.local/share/QGIS/QGIS3/profiles/default/python/plugins/SemiAutomaticClassificationPlugin/maininterface/clusteringTab.py", line 889, in kmeansIteration
                o = cfg.utls.processRaster(rD, bL, signatureList, None, cfg.utls.classification, algorithmName, oRL, oMR[0], oCR[0], previewSize, previewPoint, cfg.NoDataVal, "No", cfg.multiAddFactorsVar, cfg.bandSetsList[bandSetNumber][6])
              File "/home/mechanik/.local/share/QGIS/QGIS3/profiles/default/python/plugins/SemiAutomaticClassificationPlugin/core/utils.py", line 3957, in processRaster
                o = functionRaster(gdalBandList, signatureList, algorithmName, c, bSX, bSY, x, y, outputRasterList, outputAlgorithmRaster, outputClassificationRaster, nodataValue, macroclassCheck, previewSize, pX[lX.index(x)], pY[lY.index(y)], progressStart, progresStep, remainingBlocks, progressMessage)
              File "/home/mechanik/.local/share/QGIS/QGIS3/profiles/default/python/plugins/SemiAutomaticClassificationPlugin/core/utils.py", line 962, in classification
                if classArray == None:

2018-01-30T20:31:25 WARNING warning:/home/mechanik/.local/share/QGIS/QGIS3/profiles/default/python/plugins/SemiAutomaticClassificationPlugin/maininterface/clusteringTab.py:920: RuntimeWarning:

            invalid value encountered in true_divide

            traceback:  File "/home/mechanik/.local/share/QGIS/QGIS3/profiles/default/python/plugins/SemiAutomaticClassificationPlugin/maininterface/clusteringTab.py", line 120, in calculateClusteringAction
                self.calculateClustering()
              File "/home/mechanik/.local/share/QGIS/QGIS3/profiles/default/python/plugins/SemiAutomaticClassificationPlugin/maininterface/clusteringTab.py", line 193, in calculateClustering
                self.kmeansCalculation(rD, bL, clssOut, k_or_sigs, iterations, thresh, NoDataValue, batch, bandSetNumber)
              File "/home/mechanik/.local/share/QGIS/QGIS3/profiles/default/python/plugins/SemiAutomaticClassificationPlugin/maininterface/clusteringTab.py", line 671, in kmeansCalculation
                r, sigs, sL, distances = self.kmeansIteration(rD, bL, iteration, k_or_sigs, iterations, thresh, NoDataValue, batch, bandSetNumber)
              File "/home/mechanik/.local/share/QGIS/QGIS3/profiles/default/python/plugins/SemiAutomaticClassificationPlugin/maininterface/clusteringTab.py", line 920, in kmeansIteration
                v = cfg.rasterClustering["SUM_BAND_" + str(b) + "_c_" + str(c[0])] / cfg.rasterClustering["COUNT_BAND_" + str(b) + "_c_" + str(c[0])]

Python errors:

2018-01-30T20:31:40 WARNING Traceback (most recent call last):
              File "/home/mechanik/.local/share/QGIS/QGIS3/profiles/default/python/plugins/SemiAutomaticClassificationPlugin/maininterface/clusteringTab.py", line 774, in kmeansIteration
                k = int(k_or_sigs)
            TypeError: int() argument must be a string, a bytes-like object or a number, not 'NoneType'

            During handling of the above exception, another exception occurred:

            Traceback (most recent call last):
              File "/home/mechanik/.local/share/QGIS/QGIS3/profiles/default/python/plugins/SemiAutomaticClassificationPlugin/maininterface/clusteringTab.py", line 120, in calculateClusteringAction
                self.calculateClustering()
              File "/home/mechanik/.local/share/QGIS/QGIS3/profiles/default/python/plugins/SemiAutomaticClassificationPlugin/maininterface/clusteringTab.py", line 193, in calculateClustering
                self.kmeansCalculation(rD, bL, clssOut, k_or_sigs, iterations, thresh, NoDataValue, batch, bandSetNumber)
              File "/home/mechanik/.local/share/QGIS/QGIS3/profiles/default/python/plugins/SemiAutomaticClassificationPlugin/maininterface/clusteringTab.py", line 671, in kmeansCalculation
                r, sigs, sL, distances = self.kmeansIteration(rD, bL, iteration, k_or_sigs, iterations, thresh, NoDataValue, batch, bandSetNumber)
              File "/home/mechanik/.local/share/QGIS/QGIS3/profiles/default/python/plugins/SemiAutomaticClassificationPlugin/maininterface/clusteringTab.py", line 845, in kmeansIteration
                k = len(k_or_sigs)
            TypeError: object of type 'NoneType' has no len()

[semiautomaticgit]

Hello, thank you for reporting this. Could you also upload the SCP Log file?

[merkato]

__0semiautomaticclass.log

Thanks for the great update of SCP. And for sharing my polish tutorial on land cover classification

[semiautomaticgit]

Thank you very much. I'm glad to share your tutorial. I'll try to fix this issue.

[merkato]

I've tested ISODATA algorithm with same raster. "Use random seed signatures","Spectral Angle Mapping" settings used. Success.

Only K-means method affected.

[semiautomaticgit]

Thank you very much for your feedback. I'll look into this.

[semiautomaticgit]

Possibly it is related to system memory. Could you please try this dataset in your system https://docs.google.com/uc?id=18CXceeQrjxQoM5Z80kvlvdTI4SmVBDZn

[merkato]

This dataset is correctly processed with K-means. I will try to set more memory in settings and try again with one of Sentinel's... Thank you for investigating this.

2018-02-05 23:18 GMT+01:00 Luca Congedo notifications@github.com:

Possibly it is related to system memory. Could you please try this dataset in your system https://docs.google.com/uc?id= 18CXceeQrjxQoM5Z80kvlvdTI4SmVBDZn

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/semiautomaticgit/SemiAutomaticClassificationPlugin/issues/29#issuecomment-363240503, or mute the thread https://github.com/notifications/unsubscribe-auth/AAztvxvgXV18FHzd5WREfLcy94qqrw0Oks5tR34rgaJpZM4Ry5O7 .

-- Tomasz Nycz, Koniaków

[merkato]

Ok. Clipped Sentinel dataset was correctly processed with 4000 MB RAM set.

My proposition is to change "Memory error. Please, set a lower value of RAM in the tab Settings" as `"Memory error. Please, adjust value of RAM in the tab Settings". First information is in my opinion confusing - I was lowering RAM available to SCP, but I should set it bigger.

Once again - thank you very much for you effort to make SCP even better.

[semiautomaticgit]

Thank you very much for your feedback. Yes, I'll try to improve the memory error messages