search

maximilianh / cellBrowser

main repo: https://github.com/ucscGenomeBrowser/cellBrowser/ - Python pipeline and Javascript scatter plot library for single-cell datasets, http://cellbrowser.rtfd.org
https://github.com/ucscGenomeBrowser/cellBrowser/
GNU General Public License v3.0
102 stars 40 forks source link

Running CBBuild on Autism Data #229

Closed itskhurrams closed 2 years ago

itskhurrams commented 2 years ago

Hi, I have successfully build and run "Mini" and its working fine.

Now i download Autism Data place it in a folder. but when i try to run cbbuild on "Autism" it gives me following error. Please help. because i have to build mutiple datasets.

PS D:\CellBrowser\cellBrowser\src\Autism> python ../cbbuild --init INFO:root:Wrote D:\CellBrowser\cellBrowser\src\Autism\cellbrowser.conf INFO:root:Wrote D:\CellBrowser\cellBrowser\src\Autism\desc.conf PS D:\CellBrowser\cellBrowser\src\Autism> python ../cbbuild -o D:\CellBrowser\cellBrowser\src\public_html -p 8888 INFO:root:dataRoot is not set in ~/.cellbrowser.conf or via $CBDATAROOT. Dataset hierarchies are not supported. INFO:root:Determining if D:\CellBrowser\cellBrowser\src\public_html\sample\exprMatrix.tsv.gz needs to be created INFO:root:D:\CellBrowser\cellBrowser\src\public_html\sample\exprMatrix.tsv.gz does not exist. Must build matrix now. INFO:root:Checking and reordering meta data to D:\CellBrowser\cellBrowser\src\public_html\sample\meta.tsv INFO:root:Reading sample names from D:\CellBrowser\cellBrowser\src\Autism\meta.tsv INFO:root:Reading headers from file D:\CellBrowser\cellBrowser\src\Autism\exprMatrix.tsv.gz INFO:root:Data contains 104559 samples/cells INFO:root:Converting to numbers and compressing meta data fields WARNING:root:File colors.tsv does not exist WARNING:root:D:\CellBrowser\cellBrowser\src\Autism\acronyms.tsv specified in config file, but does not exist, skipping INFO:root:Field cell: type uniqueString, 104559 different values INFO:root:Field cluster: type enum, 17 different values INFO:root:Field sample: type enum, 41 different values INFO:root:Field individual: type int, 31 different values INFO:root:Field region: type enum, 2 different values INFO:root:Field age: type int, 14 different values INFO:root:Field sex: type enum, 2 different values INFO:root:Field diagnosis: type enum, 2 different values INFO:root:Field Capbatch: type enum, 8 different values INFO:root:Field Seqbatch: type enum, 3 different values INFO:root:Field postMinusmortemintervalhours: type int, 18 different values INFO:root:Field RNAIntegrityNumber: type float, 18 different values INFO:root:Field genes: type int, 8202 different values INFO:root:Field UMIs: type int, 20938 different values INFO:root:Field RNAmitochondrpercent: type float, 64073 different values INFO:root:Field RNAribosomalpercent: type float, 62056 different values INFO:root:Indexing meta file D:\CellBrowser\cellBrowser\src\public_html\sample\meta.tsv to D:\CellBrowser\cellBrowser\src\public_html\sample\meta.index INFO:root:Kept 104559 cells present in both meta data file and expression matrix WARNING:root:Gunzip not in PATH, falling back to Python's built-in INFO:root:Auto-detecting number type of D:\CellBrowser\cellBrowser\src\Autism\exprMatrix.tsv.gz INFO:root:Auto-detect: Numbers in matrix are of type 'float' INFO:root:Auto-detected gene IDs type: gencode-human INFO:root:Reading gene,symbol mapping from D:\CellBrowser\cellBrowser\cellbrowserData\genes\gencode-human.symbols.tsv.gz INFO:root:Copying/compressing D:\CellBrowser\cellBrowser\src\Autism\exprMatrix.tsv.gz to D:\CellBrowser\cellBrowser\src\public_html\sample\exprMatrix.tsv.gz 'cp' is not recognized as an internal or external command, operable program or batch file. ERROR:root:Could not run: cp "D:\CellBrowser\cellBrowser\src\Autism\exprMatrix.tsv.gz" "D:\CellBrowser\cellBrowser\src\public_html\sample\exprMatrix.tsv.gz" ERROR:root:Unexpected error: (<class 'SystemExit'>, SystemExit(1), <traceback object at 0x000001E5A624A900>) Traceback (most recent call last): File "D:\CellBrowser\cellBrowser\src\cbPyLib\cellbrowser\cellbrowser.py", line 4783, in cbBuildCli build(confFnames, outDir, port, redo=options.redo) File "D:\CellBrowser\cellBrowser\src\cbPyLib\cellbrowser\cellbrowser.py", line 4598, in build convertDataset(inDir, inConf, outConf, datasetDir, redo) File "D:\CellBrowser\cellBrowser\src\cbPyLib\cellbrowser\cellbrowser.py", line 3955, in convertDataset convertExprMatrix(inConf, outMatrixFname, outConf, sampleNames, geneToSym, datasetDir, needFilterMatrix) File "D:\CellBrowser\cellBrowser\src\cbPyLib\cellbrowser\cellbrowser.py", line 3285, in convertExprMatrix matType = copyMatrixTrim(matrixFname, outMatrixFname, metaSampleNames, needFilterMatrix, geneToSym, outConf, matType) File "D:\CellBrowser\cellBrowser\src\cbPyLib\cellbrowser\cellbrowser.py", line 2525, in copyMatrixTrim ret = runCommand(cmd) File "D:\CellBrowser\cellBrowser\src\cbPyLib\cellbrowser\cellbrowser.py", line 2459, in runCommand errAbort("Could not run: %s" % cmd) File "D:\CellBrowser\cellBrowser\src\cbPyLib\cellbrowser\cellbrowser.py", line 168, in errAbort sys.exit(1) SystemExit: 1

Thanks in Advance. Khurram

maximilianh commented 2 years ago

Hi Khurram,

thank you for finding this bug. I see that you're using Windows and while the output is missing a piece, I think I can guess from this output what the problem is. I forgot that the copy command is not "cp" on Windows. This is exactly why I ask in the documentation to contact us if Windows problem appear, I didn't test this enough on Windows.

I tried to fix this now, by using the "copy" command on Windows, can you re-install the software again using the "--pre" option?

pip install cellbrowser --pre

Should give you version 1.2.0a3. Then, if you re-run the cbBuild command, the file copy operation should not fail anymore.

Thanks again and let us know if this solved the problem, Max

On Sat, Oct 23, 2021 at 10:43 AM Khurram Shahzad @.***> wrote:

Hi, I have successfully build and run "Mini" and its working fine.

Now i download Autism Data place it in a folder. but when i try to run cbbuild on "Autism" it gives me following error. Please help. because i have to build mutiple datasets.

PS D:\CellBrowser\cellBrowser\src\Autism> python ../cbbuild --init INFO:root:Wrote D:\CellBrowser\cellBrowser\src\Autism\cellbrowser.conf INFO:root:Wrote D:\CellBrowser\cellBrowser\src\Autism\desc.conf PS D:\CellBrowser\cellBrowser\src\Autism> python ../cbbuild -o D:\CellBrowser\cellBrowser\src\public_html -p 8888 INFO:root:dataRoot is not set in ~/.cellbrowser.conf or via $CBDATAROOT. Dataset hierarchies are not supported. INFO:root:Determining if D:\CellBrowser\cellBrowser\src\public_html\sample\exprMatrix.tsv.gz needs to be created INFO:root:D:\CellBrowser\cellBrowser\src\public_html\sample\exprMatrix.tsv.gz does not exist. Must build matrix now. INFO:root:Checking and reordering meta data to D:\CellBrowser\cellBrowser\src\public_html\sample\meta.tsv INFO:root:Reading sample names from D:\CellBrowser\cellBrowser\src\Autism\meta.tsv INFO:root:Reading headers from file D:\CellBrowser\cellBrowser\src\Autism\exprMatrix.tsv.gz INFO:root:Data contains 104559 samples/cells INFO:root:Converting to numbers and compressing meta data fields WARNING:root:File colors.tsv does not exist WARNING:root:D:\CellBrowser\cellBrowser\src\Autism\acronyms.tsv specified in config file, but does not exist, skipping INFO:root:Field cell: type uniqueString, 104559 different values INFO:root:Field cluster: type enum, 17 different values INFO:root:Field sample: type enum, 41 different values INFO:root:Field individual: type int, 31 different values INFO:root:Field region: type enum, 2 different values INFO:root:Field age: type int, 14 different values INFO:root:Field sex: type enum, 2 different values INFO:root:Field diagnosis: type enum, 2 different values INFO:root:Field Capbatch: type enum, 8 different values INFO:root:Field Seqbatch: type enum, 3 different values INFO:root:Field postMinusmortemintervalhours: type int, 18 different values INFO:root:Field RNAIntegrityNumber: type float, 18 different values INFO:root:Field genes: type int, 8202 different values INFO:root:Field UMIs: type int, 20938 different values INFO:root:Field RNAmitochondrpercent: type float, 64073 different values INFO:root:Field RNAribosomalpercent: type float, 62056 different values INFO:root:Indexing meta file D:\CellBrowser\cellBrowser\src\public_html\sample\meta.tsv to D:\CellBrowser\cellBrowser\src\public_html\sample\meta.index INFO:root:Kept 104559 cells present in both meta data file and expression matrix WARNING:root:Gunzip not in PATH, falling back to Python's built-in INFO:root:Auto-detecting number type of D:\CellBrowser\cellBrowser\src\Autism\exprMatrix.tsv.gz INFO:root:Auto-detect: Numbers in matrix are of type 'float' INFO:root:Auto-detected gene IDs type: gencode-human INFO:root:Reading gene,symbol mapping from D:\CellBrowser\cellBrowser\cellbrowserData\genes\gencode-human.symbols.tsv.gz INFO:root:Copying/compressing D:\CellBrowser\cellBrowser\src\Autism\exprMatrix.tsv.gz to D:\CellBrowser\cellBrowser\src\public_html\sample\exprMatrix.tsv.gz 'cp' is not recognized as an internal or external command, operable program or batch file. ERROR:root:Could not run: cp "D:\CellBrowser\cellBrowser\src\Autism\exprMatrix.tsv.gz" "D:\CellBrowser\cellBrowser\src\public_html\sample\exprMatrix.tsv.gz" ERROR:root:Unexpected error: (<class 'SystemExit'>, SystemExit(1), <traceback object at 0x000001E5A624A900>) Traceback (most recent call last): File "D:\CellBrowser\cellBrowser\src\cbPyLib\cellbrowser\cellbrowser.py", line 4783, in cbBuildCli build(confFnames, outDir, port, redo=options.redo) File "D:\CellBrowser\cellBrowser\src\cbPyLib\cellbrowser\cellbrowser.py", line 4598, in build convertDataset(inDir, inConf, outConf, datasetDir, redo) File "D:\CellBrowser\cellBrowser\src\cbPyLib\cellbrowser\cellbrowser.py", line 3955, in convertDataset convertExprMatrix(inConf, outMatrixFname, outConf, sampleNames, geneToSym, datasetDir, needFilterMatrix) File "D:\CellBrowser\cellBrowser\src\cbPyLib\cellbrowser\cellbrowser.py", line 3285, in convertExprMatrix matType = copyMatrixTrim(matrixFname, outMatrixFname, metaSampleNames, needFilterMatrix, geneToSym, outConf, matType) File "D:\CellBrowser\cellBrowser\src\cbPyLib\cellbrowser\cellbrowser.py", line 2525, in copyMatrixTrim ret = runCommand(cmd) File "D:\CellBrowser\cellBrowser\src\cbPyLib\cellbrowser\cellbrowser.py", line 2459, in runCommand errAbort("Could not run: %s" % cmd) File "D:\CellBrowser\cellBrowser\src\cbPyLib\cellbrowser\cellbrowser.py", line 168, in errAbort sys.exit(1) SystemExit: 1

Thanks in Advance. Khurram

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/maximilianh/cellBrowser/issues/229, or unsubscribe https://github.com/notifications/unsubscribe-auth/AACL4TPMDHYE2XNSHHJ4O23UIJYTJANCNFSM5GSAY5SA . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

itskhurrams commented 2 years ago

Hi Max, Thank you for your support i really appreciate. Actually i am running it from code so should i download the developer branch in order to get this fixed ?

maximilianh commented 2 years ago

Yes, if you use the git repo, just do a "git pull" and then "git checkout develop". Then, re-run the cbBuild command.

Let me know if there is still a problem. I don't have a Windows machine, so my options of testing this are limited.

On Tue, Oct 26, 2021 at 9:32 AM Khurram Shahzad @.***> wrote:

Hi Max, Thank you for your support i really appreciate. Actually i am running it from code so should i download the developer branch in order to get this fixed ?

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/maximilianh/cellBrowser/issues/229#issuecomment-951641495, or unsubscribe https://github.com/notifications/unsubscribe-auth/AACL4TMXRJ74U6TMTVPN7K3UIZKPPANCNFSM5GSAY5SA . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

itskhurrams commented 2 years ago

No Luck. I downloaded the developer repo and run the command and i can see CP is still showing.

PS D:\CellBrowser\cellBrowserdevelop\src> cd .\Autism\ PS D:\CellBrowser\cellBrowserdevelop\src\Autism> python ../cbbuild --init INFO:root:Wrote D:\CellBrowser\cellBrowserdevelop\src\Autism\cellbrowser.conf INFO:root:Wrote D:\CellBrowser\cellBrowserdevelop\src\Autism\desc.conf PS D:\CellBrowser\cellBrowserdevelop\src\Autism> python ../cbbuild -o D:\CellBrowser\cellBrowserdevelop\src\public_html -p 8888 WARNING:root:The directory D:\CellBrowser\cellBrowserdevelop\src\public_html does not exist. Making a new directory now. INFO:root:dataRoot is not set in ~/.cellbrowser.conf or via $CBDATAROOT. Dataset hierarchies are not supported. INFO:root:Creating D:\CellBrowser\cellBrowserdevelop\src\public_html\sample INFO:root:Determining if D:\CellBrowser\cellBrowserdevelop\src\public_html\sample\exprMatrix.tsv.gz needs to be created INFO:root:D:\CellBrowser\cellBrowserdevelop\src\public_html\sample\exprMatrix.tsv.gz does not exist. Must build matrix now. INFO:root:Creating D:\CellBrowser\cellBrowserdevelop\src\public_html\sample\metaFields INFO:root:Checking and reordering meta data to D:\CellBrowser\cellBrowserdevelop\src\public_html\sample\meta.tsv INFO:root:Reading sample names from D:\CellBrowser\cellBrowserdevelop\src\Autism\meta.tsv INFO:root:Reading headers from file D:\CellBrowser\cellBrowserdevelop\src\Autism\exprMatrix.tsv.gz INFO:root:Data contains 104559 samples/cells INFO:root:Converting to numbers and compressing meta data fields WARNING:root:File colors.tsv does not exist WARNING:root:D:\CellBrowser\cellBrowserdevelop\src\Autism\acronyms.tsv specified in config file, but does not exist, skipping INFO:root:Field cell: type uniqueString, 104559 different values INFO:root:Field cluster: type enum, 17 different values INFO:root:Field sample: type enum, 41 different values INFO:root:Field individual: type int, 31 different values INFO:root:Field region: type enum, 2 different values INFO:root:Field age: type int, 14 different values INFO:root:Field sex: type enum, 2 different values INFO:root:Field diagnosis: type enum, 2 different values INFO:root:Field Capbatch: type enum, 8 different values INFO:root:Field Seqbatch: type enum, 3 different values INFO:root:Field postMinusmortemintervalhours: type int, 18 different values INFO:root:Field RNAIntegrityNumber: type float, 18 different values INFO:root:Field genes: type int, 8202 different values INFO:root:Field UMIs: type int, 20938 different values INFO:root:Field RNAmitochondrpercent: type float, 64073 different values INFO:root:Field RNAribosomalpercent: type float, 62056 different values INFO:root:Indexing meta file D:\CellBrowser\cellBrowserdevelop\src\public_html\sample\meta.tsv to D:\CellBrowser\cellBrowserdevelop\src\public_html\sample\meta.index INFO:root:Kept 104559 cells present in both meta data file and expression matrix WARNING:root:Gunzip not in PATH, falling back to Python's built-in INFO:root:Auto-detecting number type of D:\CellBrowser\cellBrowserdevelop\src\Autism\exprMatrix.tsv.gz INFO:root:Auto-detect: Numbers in matrix are of type 'float' INFO:root:Auto-detected gene IDs type: gencode-human INFO:root:Reading gene,symbol mapping from D:\CellBrowser\cellBrowserdevelop\cellbrowserData\genes\gencode-human.symbols.tsv.gz INFO:root:Copying/compressing D:\CellBrowser\cellBrowserdevelop\src\Autism\exprMatrix.tsv.gz to D:\CellBrowser\cellBrowserdevelop\src\public_html\sample\exprMatrix.tsv.gz 'cp' is not recognized as an internal or external command, operable program or batch file. ERROR:root:Could not run: cp "D:\CellBrowser\cellBrowserdevelop\src\Autism\exprMatrix.tsv.gz" "D:\CellBrowser\cellBrowserdevelop\src\public_html\sample\exprMatrix.tsv.gz" ERROR:root:Unexpected error: (<class 'SystemExit'>, SystemExit(1), <traceback object at 0x000001B5EDCC6DC0>) Traceback (most recent call last): File "D:\CellBrowser\cellBrowserdevelop\src\cbPyLib\cellbrowser\cellbrowser.py", line 4783, in cbBuildCli build(confFnames, outDir, port, redo=options.redo) File "D:\CellBrowser\cellBrowserdevelop\src\cbPyLib\cellbrowser\cellbrowser.py", line 4598, in build convertDataset(inDir, inConf, outConf, datasetDir, redo) File "D:\CellBrowser\cellBrowserdevelop\src\cbPyLib\cellbrowser\cellbrowser.py", line 3955, in convertDataset convertExprMatrix(inConf, outMatrixFname, outConf, sampleNames, geneToSym, datasetDir, needFilterMatrix) File "D:\CellBrowser\cellBrowserdevelop\src\cbPyLib\cellbrowser\cellbrowser.py", line 3285, in convertExprMatrix matType = copyMatrixTrim(matrixFname, outMatrixFname, metaSampleNames, needFilterMatrix, geneToSym, outConf, matType) File "D:\CellBrowser\cellBrowserdevelop\src\cbPyLib\cellbrowser\cellbrowser.py", line 2525, in copyMatrixTrim ret = runCommand(cmd) File "D:\CellBrowser\cellBrowserdevelop\src\cbPyLib\cellbrowser\cellbrowser.py", line 2459, in runCommand errAbort("Could not run: %s" % cmd) File "D:\CellBrowser\cellBrowserdevelop\src\cbPyLib\cellbrowser\cellbrowser.py", line 168, in errAbort sys.exit(1) SystemExit: 1 PS D:\CellBrowser\cellBrowserdevelop\src\Autism>

itskhurrams commented 2 years ago

image

maximilianh commented 2 years ago

Yes, somehow you still have the old version of the file. Line 2525 of cellbrowser.py on your disk has the old line, ret = runCommand(cmd), but the version of the file has these lines:

    if inFname.endswith(".gz"):
        shutil.copyfile(inFname, outFname)
    else:
        if platform.system()=="Windows":
            # slow but quick hack for Github #229
            tmpFname = outFname+".tmp"
            shutil.copyfile(inFname, tmpFname)
            runGzip(tmpFname, outFname)

Somehow D:\CellBrowser/cellBrowserdevelop is not the "develop" branch or didn't get updated from Github.

Can you try git pull in that directory, "git checkout develop", check that you're on develop using "git branch" and then check line 2525 in cellbrowser.py ?

On Tue, Oct 26, 2021 at 10:08 AM Khurram Shahzad @.***> wrote:

[image: image] https://user-images.githubusercontent.com/5083361/138835286-b67550ff-927e-4e60-8267-1ea452e63a37.png

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/maximilianh/cellBrowser/issues/229#issuecomment-951668316, or unsubscribe https://github.com/notifications/unsubscribe-auth/AACL4TNSZA2JZ6QCFLLMM7LUIZOX7ANCNFSM5GSAY5SA . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

maximilianh commented 2 years ago

Thanks for closing the ticket. I assume that this means that my last commit solved your problem?

Let us know if you run into something else. I admit that I haven't tested on windows, but that was a long time ago.

On Thu, Oct 28, 2021 at 2:37 PM Khurram Shahzad @.***> wrote:

Closed #229 https://github.com/maximilianh/cellBrowser/issues/229.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/maximilianh/cellBrowser/issues/229#event-5533817559, or unsubscribe https://github.com/notifications/unsubscribe-auth/AACL4TLF7LO4MZWG7PPKOF3UJE7W3ANCNFSM5GSAY5SA . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.