search

Ecogenomics / CheckM

Assess the quality of microbial genomes recovered from isolates, single cells, and metagenomes
https://ecogenomics.github.io/CheckM/
GNU General Public License v3.0
336 stars 73 forks source link

Issue 3770 #309

Closed howla1ke closed 3 years ago

howla1ke commented 3 years ago

Hi, Has anyone had this issue and resolved it?

I used this command: [xxxxxxx@dev-amd20-v100 CheckM_test]$ checkm analyze Bacteria.marker.file batch_bins Mega_batch_bins_checkm/ -x fasta

Data root path: /opt/software/CheckM/1.1.3-intel-2020a-Python-3.8.2/data manifestFile: /opt/software/CheckM/1.1.3-intel-2020a-Python-3.8.2/data/.dmanifest [2021-07-20 15:18:24] INFO: CheckM v1.1.3 [2021-07-20 15:18:24] INFO: checkm analyze Bacteria.marker.file batch_bins Mega_batch_bins_checkm/ -x fasta [2021-07-20 15:18:24] INFO: [CheckM - analyze] Identifying marker genes in bins. [2021-07-20 15:18:58] INFO: Identifying marker genes in 106 bins with 1 threads:

Unexpected error: <class 'ImportError'> Traceback (most recent call last): File "/opt/software/Python/3.8.2-GCCcore-9.3.0/lib/python3.8/multiprocessing/synchronize.py", line 28, in from _multiprocessing import SemLock, sem_unlink ImportError: cannot import name 'SemLock' from '_multiprocessing' (/opt/software/Python/3.8.2-GCCcore-9.3.0/lib/python3.8/lib-dynload/_multiprocessing.cpython-38-x86_64-linux-gnu.so)

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/opt/software/CheckM/1.1.3-intel-2020a-Python-3.8.2/bin/checkm", line 611, in checkmParser.parseOptions(args) File "/opt/software/CheckM/1.1.3-intel-2020a-Python-3.8.2/lib/python3.8/site-packages/checkm/main.py", line 846, in parseOptions self.analyze(options) File "/opt/software/CheckM/1.1.3-intel-2020a-Python-3.8.2/lib/python3.8/site-packages/checkm/main.py", line 269, in analyze binIdToModels = mgf.find(binFiles, File "/opt/software/CheckM/1.1.3-intel-2020a-Python-3.8.2/lib/python3.8/site-packages/checkm/markerGeneFinder.py", line 58, in find workerQueue = mp.Queue() File "/opt/software/Python/3.8.2-GCCcore-9.3.0/lib/python3.8/multiprocessing/context.py", line 103, in Queue return Queue(maxsize, ctx=self.get_context()) File "/opt/software/Python/3.8.2-GCCcore-9.3.0/lib/python3.8/multiprocessing/queues.py", line 39, in init from .synchronize import SEM_VALUE_MAX as maxsize File "/opt/software/Python/3.8.2-GCCcore-9.3.0/lib/python3.8/multiprocessing/synchronize.py", line 30, in raise ImportError("This platform lacks a functioning sem_open" + ImportError: This platform lacks a functioning sem_open implementation, therefore, the required synchronization primitives needed will not function, see issue 3770.

donovan-h-parks commented 3 years ago

Hi. I haven't seen this error before. The error message suggests your platform is missing so a sem implementation which I assume standard for semaphore. From your directory names, it appears like you might be using a non-standard version of Python (made and/or compiled by Intel?).

howla1ke commented 3 years ago

Hi,

The first error I had was this:

checkm data setRoot /mnt/gs18/scratch/users/howla1keCMICH/SAMtools/CheckM_test

It seems that the CheckM data folder has not been set yet or has been removed. Please run 'checkm data setRoot'.

Traceback (most recent call last):

File "/opt/software/CheckM/1.1.3-intel-2020a-Python-3.8.2/lib/python3.8/site-packages/checkm/checkmData.py", line 87, in checkPermissions

open(self.configFile, 'a')

PermissionError: [Errno 13] Permission denied: '/opt/software/CheckM/1.1.3-intel-2020a-Python-3.8.2/lib/python3.8/site-packages/checkm/DATA_CONFIG'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):

File "/opt/software/CheckM/1.1.3-intel-2020a-Python-3.8.2/bin/checkm", line 36, in 

from checkm import main

File "/opt/software/CheckM/1.1.3-intel-2020a-Python-3.8.2/lib/python3.8/site-packages/checkm/main.py", line 25, in 

from checkm.defaultValues import DefaultValues

File "/opt/software/CheckM/1.1.3-intel-2020a-Python-3.8.2/lib/python3.8/site-packages/checkm/defaultValues.py", line 26, in 

class DefaultValues():

File "/opt/software/CheckM/1.1.3-intel-2020a-Python-3.8.2/lib/python3.8/site-packages/checkm/defaultValues.py", line 29, in DefaultValues

__DBM = DBManager()

File "/opt/software/CheckM/1.1.3-intel-2020a-Python-3.8.2/lib/python3.8/site-packages/checkm/checkmData.py", line 120, in init

if not self.setRoot():

File "/opt/software/CheckM/1.1.3-intel-2020a-Python-3.8.2/lib/python3.8/site-packages/checkm/checkmData.py", line 142, in setRoot

if not self.config.checkPermissions():

File "/opt/software/CheckM/1.1.3-intel-2020a-Python-3.8.2/lib/python3.8/site-packages/checkm/checkmData.py", line 88, in checkPermissions

except (IOError, e):

NameError: name 'e' is not defined

@.*** CheckM_test]$ checkm

It seems that the CheckM data folder has not been set yet or has been removed. Please run 'checkm data setRoot'.

Traceback (most recent call last):

File "/opt/software/CheckM/1.1.3-intel-2020a-Python-3.8.2/lib/python3.8/site-packages/checkm/checkmData.py", line 87, in checkPermissions

open(self.configFile, 'a')

PermissionError: [Errno 13] Permission denied: '/opt/software/CheckM/1.1.3-intel-2020a-Python-3.8.2/lib/python3.8/site-packages/checkm/DATA_CONFIG'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):

File "/opt/software/CheckM/1.1.3-intel-2020a-Python-3.8.2/bin/checkm", line 36, in 

from checkm import main

File "/opt/software/CheckM/1.1.3-intel-2020a-Python-3.8.2/lib/python3.8/site-packages/checkm/main.py", line 25, in 

from checkm.defaultValues import DefaultValues

File "/opt/software/CheckM/1.1.3-intel-2020a-Python-3.8.2/lib/python3.8/site-packages/checkm/defaultValues.py", line 26, in 

class DefaultValues():

File "/opt/software/CheckM/1.1.3-intel-2020a-Python-3.8.2/lib/python3.8/site-packages/checkm/defaultValues.py", line 29, in DefaultValues

__DBM = DBManager()

File "/opt/software/CheckM/1.1.3-intel-2020a-Python-3.8.2/lib/python3.8/site-packages/checkm/checkmData.py", line 120, in init

if not self.setRoot():

File "/opt/software/CheckM/1.1.3-intel-2020a-Python-3.8.2/lib/python3.8/site-packages/checkm/checkmData.py", line 142, in setRoot

if not self.config.checkPermissions():

File "/opt/software/CheckM/1.1.3-intel-2020a-Python-3.8.2/lib/python3.8/site-packages/checkm/checkmData.py", line 88, in checkPermissions

except (IOError, e):

NameError: name 'e' is not defined

@.*** CheckM_test]$ checkm -h

It seems that the CheckM data folder has not been set yet or has been removed. Please run 'checkm data setRoot'.

Traceback (most recent call last):

File "/opt/software/CheckM/1.1.3-intel-2020a-Python-3.8.2/lib/python3.8/site-packages/checkm/checkmData.py", line 87, in checkPermissions

open(self.configFile, 'a')

PermissionError: [Errno 13] Permission denied: '/opt/software/CheckM/1.1.3-intel-2020a-Python-3.8.2/lib/python3.8/site-packages/checkm/DATA_CONFIG'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):

File "/opt/software/CheckM/1.1.3-intel-2020a-Python-3.8.2/bin/checkm", line 36, in 

from checkm import main

File "/opt/software/CheckM/1.1.3-intel-2020a-Python-3.8.2/lib/python3.8/site-packages/checkm/main.py", line 25, in 

from checkm.defaultValues import DefaultValues

File "/opt/software/CheckM/1.1.3-intel-2020a-Python-3.8.2/lib/python3.8/site-packages/checkm/defaultValues.py", line 26, in 

class DefaultValues():

File "/opt/software/CheckM/1.1.3-intel-2020a-Python-3.8.2/lib/python3.8/site-packages/checkm/defaultValues.py", line 29, in DefaultValues

__DBM = DBManager()

File "/opt/software/CheckM/1.1.3-intel-2020a-Python-3.8.2/lib/python3.8/site-packages/checkm/checkmData.py", line 120, in init

if not self.setRoot():

File "/opt/software/CheckM/1.1.3-intel-2020a-Python-3.8.2/lib/python3.8/site-packages/checkm/checkmData.py", line 142, in setRoot

if not self.config.checkPermissions():

File "/opt/software/CheckM/1.1.3-intel-2020a-Python-3.8.2/lib/python3.8/site-packages/checkm/checkmData.py", line 88, in checkPermissions

except (IOError, e):

NameError: name 'e' is not defined

​The HPCC helpdesk created this patch:

Hello, The issue has been reported in the github page on Jul 23, 2020:

https://github.com/Ecogenomics/CheckM/issues/270

In the final response, a user provided an alternative way to change the data root. I applied the patch to the version of CheckM/1.1.3-Python-3.8.2. Please use the command:

ml -* intel/2020a CheckM/1.1.3-Python-3.8.2

to load the module. By default, the data root is /opt/software/CheckM/1.1.3-intel-2020a-Python-3.8.2/data

If you would like to change it, you can set the environment variable CHECKM_DATA_ROOT to be the directory you would like, for example:

export CHECKM_DATA_ROOT=/mnt/gs18/scratch/users/xxxxx/SAMtools/CheckM_test

instead of using checkm data setRoot command. After this is set, you can use the command:

checkm

to see the change of data root in the beginning of the output:

Data root path: /mnt/gs18/scratch/users/howla1keCMICH/SAMtools/CheckM_test manifestFile: /opt/software/CheckM/1.1.3-intel-2020a-Python-3.8.2/data/.dmanifest

...::: CheckM v1.1.3 :::...

I installed per there instructions and that is when the new error was generated! Any advice you can give me would be appreciated. I am brand new to bioinformatics!

Thanks, Katie

Katie Howland Graduate Student; PhD Earth & Ecosystem Science [cid:05adacea-9692-44f9-a9b8-0b156174363b]

From: Donovan Parks @.> Sent: Tuesday, July 20, 2021 5:39 PM To: Ecogenomics/CheckM @.> Cc: Howland, Katie Ellen @.>; Author @.> Subject: [External] Re: [Ecogenomics/CheckM] Issue 3770 (#309)

Hi. I haven't seen this error before. The error message suggests your platform is missing so a sem implementation which I assume standard for semaphore. From your directory names, it appears like you might be using a non-standard version of Python (made and/or compiled by Intel?).

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHubhttps://github.com/Ecogenomics/CheckM/issues/309#issuecomment-883725836, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AUQWWTDXAWCHR6IG3XLCNPLTYXUK3ANCNFSM5AWN7LAA.

donovan-h-parks commented 3 years ago

This error PermissionError: [Errno 13] Permission denied: '/opt/software/CheckM/1.1.3-intel-2020a-Python-3.8.2/lib/python3.8/site-packages/checkm/DATA_CONFIG' indicates you don't have permission to write to the specified directory. In general, you should download the required CheckM data and place it in a directory you have permission to read and write to. Otherwise, your system admin may be able to setup CheckM for more broad use.

https://github.com/Ecogenomics/CheckM/wiki/Installation#how-to-install-checkm

howla1ke commented 3 years ago

Hi,

I do have permission to write to this directory! I even ensured by using the "chmod +rwx ~/directory" command.

@.*** CheckM_test]$ chmod +rwx /mnt/gs18/scratch/users/howla1keCMICH/SAMtools/CheckM_test

@.*** CheckM_test]$ checkm data setRoot ~/CheckM_test

Data root path: /opt/software/CheckM/1.1.3-intel-2020a-Python-3.8.2/data

manifestFile: /opt/software/CheckM/1.1.3-intel-2020a-Python-3.8.2/data/.dmanifest

[2021-07-20 18:29:35] INFO: CheckM v1.1.3

[2021-07-20 18:29:35] INFO: checkm data setRoot /mnt/home/howla1keCMICH/CheckM_test

[2021-07-20 18:29:35] INFO: [CheckM - data] Check for database updates. [setRoot]

Unexpected error: <class 'NameError'>

Traceback (most recent call last):

File "/opt/software/CheckM/1.1.3-intel-2020a-Python-3.8.2/lib/python3.8/site-packages/checkm/checkmData.py", line 89, in checkPermissions

open(self.configFile, 'a')

PermissionError: [Errno 13] Permission denied: '/opt/software/CheckM/1.1.3-intel-2020a-Python-3.8.2/lib/python3.8/site-packages/checkm/DATA_CONFIG'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):

File "/opt/software/CheckM/1.1.3-intel-2020a-Python-3.8.2/bin/checkm", line 611, in 

checkmParser.parseOptions(args)

File "/opt/software/CheckM/1.1.3-intel-2020a-Python-3.8.2/lib/python3.8/site-packages/checkm/main.py", line 834, in parseOptions

self.updateCheckM_DB(options)

File "/opt/software/CheckM/1.1.3-intel-2020a-Python-3.8.2/lib/python3.8/site-packages/checkm/main.py", line 78, in updateCheckM_DB

DBM = DBManager(set_path=action[1])

File "/opt/software/CheckM/1.1.3-intel-2020a-Python-3.8.2/lib/python3.8/site-packages/checkm/checkmData.py", line 109, in init

self.setRoot(set_path)

File "/opt/software/CheckM/1.1.3-intel-2020a-Python-3.8.2/lib/python3.8/site-packages/checkm/checkmData.py", line 148, in setRoot

if not self.config.checkPermissions():

File "/opt/software/CheckM/1.1.3-intel-2020a-Python-3.8.2/lib/python3.8/site-packages/checkm/checkmData.py", line 90, in checkPermissions

except (IOError, e):

NameError: name 'e' is not defined

Katie Howland Graduate Student; PhD Earth & Ecosystem Science [cid:9e0a37da-2294-4ade-87aa-67711dd644d7]

From: Donovan Parks @.> Sent: Tuesday, July 20, 2021 6:16 PM To: Ecogenomics/CheckM @.> Cc: Howland, Katie Ellen @.>; Author @.> Subject: [External] Re: [Ecogenomics/CheckM] Issue 3770 (#309)

This error PermissionError: [Errno 13] Permission denied: '/opt/software/CheckM/1.1.3-intel-2020a-Python-3.8.2/lib/python3.8/site-packages/checkm/DATA_CONFIG' indicates you don't have permission to write to the specified directory. In general, you should download the required CheckM data and place it in a directory you have permission to read and write to. Otherwise, your system admin may be able to setup CheckM for more broad use.

https://github.com/Ecogenomics/CheckM/wiki/Installation#how-to-install-checkm

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHubhttps://github.com/Ecogenomics/CheckM/issues/309#issuecomment-883742965, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AUQWWTFMXVHDLTC354CHTBTTYXYVHANCNFSM5AWN7LAA.

donovan-h-parks commented 3 years ago

Hi. I'd try running the command with sudo if you have admin privileges. Otherwise, perhaps you can ask your sys admin. I'm not really sure what else to suggest. Sorry I can't be of more direct help.

donovan-h-parks commented 3 years ago

Closing as inactive for >2 weeks.