Semiautomatic Classification Plugin version 7 doesn't work in QGIS 3.16.1

[cesaregal]

Mac User MacOS 10.13.6 High Sierra. In QGIS 3.16.1 Semiautomatic Classification Plugin version 6 works while Semiautomatic Classification Plugin version 7 doesn't work. Perhaps it is due to the use of multiprocessing in version 7. SCP version 6 doesn't use multiprocessing while version 7 do. (SCP version 7 works both in Linux and Windows operating systems). Warning: tbl_srs creation failed in qgis.db. Database is locked.

Describe the bug

How to Reproduce

1. Go to '...'
2. Click on '....'
3. Scroll down to '....'
4. See error -->

QGIS and OS versions

Additional context

[gioman]

@cesaregal you must report to the plugin author.

[semiautomaticgit]

Hello @gioman , I think it is a QGIS issue related to multiprocessing implementation in Mac. Please, look into it.

[gioman]

Please, look into it.

I guess that we need @PeterPetrik word here.

[Fangsheng-Zhou]

Thank you for uploading this issue! I am running the newest os, Big Sur, and the same problem happened!

[PeterPetrik]

I really do not understand what is meant by "QGIS issue related to multiprocessing implementation in Mac".

As @gioman said, please raise the bug with the developer of the plugin https://plugins.qgis.org/plugins/SemiAutomaticClassificationPlugin or seek one of the commertial support https://qgis.org/en/site/forusers/commercial_support.html

If there is a problem with QGIS, please extract the minimal pyqgis example that crashes QGIS or shows an internal problem in QGIS libraries and raise the ticket in this tracker.

[semiautomaticgit]

I really do not understand what is meant by "QGIS issue related to multiprocessing implementation in Mac".

As @gioman said, please raise the bug with the developer of the plugin https://plugins.qgis.org/plugins/SemiAutomaticClassificationPlugin or seek one of the commertial support https://qgis.org/en/site/forusers/commercial_support.html

If there is a problem with QGIS, please extract the minimal pyqgis example that crashes QGIS or shows an internal problem in QGIS libraries and raise the ticket in this tracker.

Hello @PeterPetrik ,

I'm the developer of the https://plugins.qgis.org/plugins/SemiAutomaticClassificationPlugin .

I think this issue is related to Python Multiprocessing in Mac, because the same function works in Linux and Windows operating systems. Of course, it doesn't mean it is a QGIS issue, it could be something related to Python implementation.

Do you know if there are known issues related to Multiprocessing with QGIS?

I've received a few tickets related to the plugin in Mac OS (for instance see https://github.com/semiautomaticgit/SemiAutomaticClassificationPlugin/issues/144), and I copy here a traceback as example: 2020-12-04T22:01:19 WARNING Traceback (most recent call last): File "/Users/rahulgour/Library/Application Support/QGIS/QGIS3/profiles/RahulK/python/plugins/SemiAutomaticClassificationPlugin/maininterface/clipmultiplerasters.py", line 82, in clipRastersAction self.clipRasters() File "/Users/rahulgour/Library/Application Support/QGIS/QGIS3/profiles/RahulK/python/plugins/SemiAutomaticClassificationPlugin/maininterface/clipmultiplerasters.py", line 223, in clipRasters cfg.utls.GDALCopyRaster(vrtCheck, f, 'GTiff', cfg.rasterCompression, 'LZW') File "/Users/rahulgour/Library/Application Support/QGIS/QGIS3/profiles/RahulK/python/plugins/SemiAutomaticClassificationPlugin/core/utils.py", line 7340, in GDALCopyRaster manager = cfg.MultiManagerSCP() File "/Applications/QGIS3.10.app/Contents/Frameworks/Python.framework/Versions/Current/lib/python3.7/multiprocessing/context.py", line 56, in Manager m.start() File "/Applications/QGIS3.10.app/Contents/Frameworks/Python.framework/Versions/Current/lib/python3.7/multiprocessing/managers.py", line 547, in start self._address = reader.recv() File "/Applications/QGIS3.10.app/Contents/Frameworks/Python.framework/Versions/Current/lib/python3.7/multiprocessing/connection.py", line 250, in recv buf = self._recv_bytes() File "/Applications/QGIS3.10.app/Contents/Frameworks/Python.framework/Versions/Current/lib/python3.7/multiprocessing/connection.py", line 407, in _recv_bytes buf = self._recv(4) File "/Applications/QGIS3.10.app/Contents/Frameworks/Python.framework/Versions/Current/lib/python3.7/multiprocessing/connection.py", line 383, in _recv raise EOFError EOFError

Unfortunately I don't have a Mac for testing, so hopefully @cesaregal can provide you the information you need.

Thank you very much!

[semiautomaticgit]

Possibly this issue is related to https://github.com/qgis/QGIS/issues/28697 , because several Mac users reported that QGIS opened additional QGIS instances using the plugin functions that use multiprocessing. Thank you for your time

[PeterPetrik]

Can you please try with qgis 3.16.1? QGIS 3.10 is soon out of support and also we completely refactored qgis packaging for 3.16.1.

[semiautomaticgit]

Thank you @PeterPetrik , I'll also try to replicate the plugin function in a simple script to be tested by Mac users, so it could be easier to analyze

[rahulkgour]

**As discussed with @semiautomaticgit, mentioning my observations below;

Today I have tried with QGIS 3.16.1, where I was using QGIS 3.10 earlier. I have received the below error and the Raster clipping process got stuck with no output.**

2020-12-12T23:44:08 WARNING Traceback (most recent call last): File "/Users/rahulgour/Library/Application Support/QGIS/QGIS3/profiles/RahulK/python/plugins/SemiAutomaticClassificationPlugin/maininterface/clipmultiplerasters.py", line 82, in clipRastersAction self.clipRasters() File "/Users/rahulgour/Library/Application Support/QGIS/QGIS3/profiles/RahulK/python/plugins/SemiAutomaticClassificationPlugin/maininterface/clipmultiplerasters.py", line 223, in clipRasters cfg.utls.GDALCopyRaster(vrtCheck, f, 'GTiff', cfg.rasterCompression, 'LZW') File "/Users/rahulgour/Library/Application Support/QGIS/QGIS3/profiles/RahulK/python/plugins/SemiAutomaticClassificationPlugin/core/utils.py", line 6923, in GDALCopyRaster manager = cfg.MultiManagerSCP() File "/Applications/QGIS.app/Contents/MacOS/../Resources/python/multiprocessing/context.py", line 56, in Manager m.start() File "/Applications/QGIS.app/Contents/MacOS/../Resources/python/multiprocessing/managers.py", line 567, in start self._address = reader.recv() File "/Applications/QGIS.app/Contents/MacOS/../Resources/python/multiprocessing/connection.py", line 250, in recv buf = self._recv_bytes() File "/Applications/QGIS.app/Contents/MacOS/../Resources/python/multiprocessing/connection.py", line 407, in _recv_bytes buf = self._recv(4) File "/Applications/QGIS.app/Contents/MacOS/../Resources/python/multiprocessing/connection.py", line 383, in _recv raise EOFError EOFError

[cesaregal]

Then @gioman, @PeterPetrik and @semiautomaticgit I confirm, as mentioned by others in other posts referenced above, that, when I'm working in QGIS Version 3.16.1 with Semiautomatic Classification Plugin Version 7, QGIS open multiple new windows and it becomes even impossible to stop the work indefinitely in progress; you have to turn off the computer. I enclose a screenshot showing the opening of two new QGIS windows and a screenshot with SCP log file. I remain at your disposal for any clarification. Thank you for your interest in solving the problem Cesare

[semiautomaticgit]

Hello @cesaregal , I have included a multiprocess test in the plugin that doesn't required input data. Please click the button "Test dependencies" in the tab Debug (https://semiautomaticclassificationmanual.readthedocs.io/en/latest/faq.html#how-can-i-report-an-error) and check if you get the same issue.

[Fangsheng-Zhou]

@cesaregal Hey Cesare, I think @semiautomaticgit Luca is right, it is the issue of multiprocessing. I installed windows via boot camp and test the windows version of QGIS, everything runs smoothly. If you have enough spaces on your mac system, prob considers installing windows just in case. I guess not a lot of people using QGIS on MacOS, so it cannot be properly tested.

[cesaregal]

Hello @semiautomaticgit , I've clicked the button "Test dependencies" in the tab Debug and I had the same error. Three new QGIS windows have opened and the system is no longer responding.

[semiautomaticgit]

Hello @semiautomaticgit , I've clicked the button "Test dependencies" in the tab Debug and I had the same error. Three new QGIS windows have opened and the system is no longer responding.

Thank you very much. If @PeterPetrik could be so kind to test it, maybe he can say if it is a QGIS issue or something that I can fix in the plugin.

[PeterPetrik]

I can see the problem, 3 new QGIS instances and if I close one others are spawning.

I see errors like

2021-01-01T15:15:00     CRITICAL    Invalid Data Source : /-c is not a valid or recognized data source.
2021-01-01T15:15:00     CRITICAL    Invalid Data Source : /from multiprocessing.spawn import spawn_main; spawn_main(tracker_fd=36, pipe_handle=47) is not a valid or recognized data source.
2021-01-01T15:15:00     CRITICAL    Invalid Data Source : /--multiprocessing-fork is not a valid or recognized data source.

in the console.

[PeterPetrik]

I am sorry, to find if it is QGIS core/packaging or your plugin/python multiprocessing issue, it would take me a significant time (day(s)) due to quite complex plugin and all the dependancies all the way

to solve the issue, I would need a simplest possible pure-python code that I can copy-paste to QGIS python console that triggers the problem.

if you can create such python-code that shows the problem with the packagins or QGIS support for it, it would help a great deal to solve the issue.

[semiautomaticgit]

Thank you very much @PeterPetrik .

I've tried to create a simple script that at least shows you how the multiprocess is triggered in the plugin.

import multiprocessing as mp
from multiprocessing import Pool, Manager
import time

manager = Manager()

# interprocess
def processRasterDev(writerLog = None):
    import os
    import sys
    import inspect
    import time
    import datetime
    import random
    import numpy as np
    try:
        from scipy.ndimage import label
    except:
        pass
    try:
        import scipy.stats.distributions as statdistr
    except:
        pass
    try:
        from scipy import signal
    except:
        pass
    from osgeo import gdal
    from osgeo import ogr
    from osgeo import osr

    wrtProc = str(writerLog[0])
    progressQueue = writerLog[1]

    # function
    for i in range(0, 5):
        time.sleep(1)
    return 'OK'

threadNumber = 3
# progress queue
pMQ = manager.Queue()

# parts of the calculation
sez = list(range(0, 10, threadNumber)) 
sez.append(10)

for s in range(0, len(sez)-1):
    # pool
    pool = Pool(processes=len(list(range(sez[s], sez[s+1]))))
    results = []
    for p in range(sez[s], sez[s+1]):
        wrtP = [p, pMQ]
        c = pool.apply_async(processRasterDev, args=(wrtP))
        results.append([c, p])
    # update progress
    while True:
        pR = []
        for r in results:
            pR.append(r[0].ready())
        if all(pR):
            break
        time.sleep(1)
        try:
            pMQp = pMQ.get(False)
            pgQ = int(pMQp[0])
            print(pgQ)
        except:
            print('...')

    # get results
    subprocRes = {}
    for r in results:
        res = r[0].get()
        subprocRes[r[1]] = res[0]
        if len(str(res[1])) > 0:
            pool.close()
            pool.terminate()
            print('No')
    pool.close()
    pool.terminate()
    print(subprocRes)

Unfortunately if I run the script in a QGIS console it throws the following error:

Traceback (most recent call last):
  File "/usr/lib/python3.7/code.py", line 90, in runcode
    exec(code, self.locals)
  File "<input>", line 1, in <module>
  File "<string>", line 76, in <module>
  File "/usr/lib/python3.7/multiprocessing/pool.py", line 657, in get
    raise self._value
  File "/usr/lib/python3.7/multiprocessing/pool.py", line 431, in _handle_tasks
    put(task)
  File "/usr/lib/python3.7/multiprocessing/connection.py", line 206, in send
    self._send_bytes(_ForkingPickler.dumps(obj))
  File "/usr/lib/python3.7/multiprocessing/reduction.py", line 51, in dumps
    cls(buf, protocol).dump(obj)
_pickle.PicklingError: Can't pickle <function processRasterDev at 0x7fd689468ae8>: import of module '__console__' failed

I don't know if there is a proper way to call multiprocessing through the QGIS console, but I think it is different from the plugin version where it works. Possibly it is because of the module console?

Thank you and I wish you a very happy new year!

[semiautomaticgit]

I've done a little research (unfortunately I don't have a Mac for testing). Possibly the issue could be related to the fact that sys.executable returns the path to QGIS and multiprocessing uses this path instead of Python path (see this). This could explain the new QGIS instances.

I've updated the plugin trying to set the executable path to Python. @cesaregal @rahulkgour @jzhou0222 please update the plugin to version 7.3.4 and test it again. Thanks

[Fangsheng-Zhou]

@semiautomaticgit Howdy Luca! I just tested quick, still, some fails appear. I'm running the latest 3.16.2. The good thing is QGIS won't restart with multiple windows. I've attached a screenshot of the error below.

[semiautomaticgit]

Thank you @jzhou0222 , I think we are on the right path. If so, I think it is not a QGIS issue but something I can solve on the plugin side.

Could you please open the QGIS Python console and run the following script returning the output?

import os
import sys

dPref = os.environ['PATH'].split(':')
print(dPref)
for flPref in dPref:
    flPrefPy = os.path.join(flPref, 'python3')
    if os.path.isfile(flPrefPy):
        print(flPrefPy)

It should show information about Python path.

[Fangsheng-Zhou]

@semiautomaticgit That's what I got ⬇️

[semiautomaticgit]

Thank you very much! It seems that the path to Python is correctly found. About the fail I would need the SCP log file. I think we can continue this here https://github.com/semiautomaticgit/SemiAutomaticClassificationPlugin/issues/144 In the meantime many thanks to @PeterPetrik