Sample Dataset TypeError: 'NoneType'

[apblair]

Running into an error when attempting to build a viewer with sample dataset in sampleData/sample1.

Traceback (most recent call last): File "../../src/cbBuild", line 10, in cellbrowser.convertAndCopyCli() File "../../src/cbPyLib/cellbrowser.py", line 2145, in convertAndCopyCli convertAndCopy(confFnames, outDir, port) File "../../src/cbPyLib/cellbrowser.py", line 2119, in convertAndCopy convertDataset(inConf, outConf, datasetDir) File "../../src/cbPyLib/cellbrowser.py", line 1962, in convertDataset convertExprMatrix(inConf, outMatrixFname, outConf, sampleNames, geneToSym, datasetDir, needFilterMatrix) File "../../src/cbPyLib/cellbrowser.py", line 1696, in convertExprMatrix matType = matrixToBin(outMatrixFname, geneToSym, binMat, binMatIndex, discretBinMat, discretMatrixIndex) File "../../src/cbPyLib/cellbrowser.py", line 1035, in matrixToBin matType, sampleNames = matReader.open(fname) TypeError: 'NoneType' object is not iterable

[maximilianh]

Thanks for this report!

I just did a fresh git clone into an empty directory, then did:

cd sampleData/sample1/ ../../src/cbBuild

and it worked fine. Maybe I did something in the my commits over the last few days that fixed your problem? Could you do a "git pull" and try again?

On Fri, Sep 7, 2018 at 8:13 PM, Andrew Blair notifications@github.com wrote:

Running into an error when attempting to build a viewer with sample dataset in sampleData/sample1.

Traceback (most recent call last): File "../../src/cbBuild", line 10, in

cellbrowser.convertAndCopyCli() File "../../src/cbPyLib/cellbrowser.py", line 2145, in convertAndCopyCli convertAndCopy(confFnames, outDir, port) File "../../src/cbPyLib/cellbrowser.py", line 2119, in convertAndCopy convertDataset(inConf, outConf, datasetDir) File "../../src/cbPyLib/cellbrowser.py", line 1962, in convertDataset convertExprMatrix(inConf, outMatrixFname, outConf, sampleNames, geneToSym, datasetDir, needFilterMatrix) File "../../src/cbPyLib/cellbrowser.py", line 1696, in convertExprMatrix matType = matrixToBin(outMatrixFname, geneToSym, binMat, binMatIndex, discretBinMat, discretMatrixIndex) File "../../src/cbPyLib/cellbrowser.py", line 1035, in matrixToBin matType, sampleNames = matReader.open(fname) TypeError: 'NoneType' object is not iterable — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub, or mute the thread.

[apblair]

After pulling the new commits, the sample data is running into a JSON decode error now.

cd sampleData/sample1/ ../../src/cbBuild -o ~/cells

Traceback (most recent call last): File "../../src/cbBuild", line 10, in cellbrowser.convertAndCopyCli() File "../../src/cbPyLib/cellbrowser.py", line 2246, in convertAndCopyCli convertAndCopy(confFnames, outDir, port) File "../../src/cbPyLib/cellbrowser.py", line 2220, in convertAndCopy convertDataset(inConf, outConf, datasetDir) File "../../src/cbPyLib/cellbrowser.py", line 2056, in convertDataset doMatrix = matrixOrSamplesHaveChanged(datasetDir, inMatrixFname, outMatrixFname, outConf) File "../../src/cbPyLib/cellbrowser.py", line 2001, in matrixOrSamplesHaveChanged lastConf = json.load(open(confName)) File "/soe/apblair/anaconda3/envs/single_cell/lib/python3.5/json/init.py", line 268, in load parse_constant=parse_constant, object_pairs_hook=object_pairs_hook, **kw) File "/soe/apblair/anaconda3/envs/single_cell/lib/python3.5/json/init.py", line 319, in loads return _default_decoder.decode(s) File "/soe/apblair/anaconda3/envs/single_cell/lib/python3.5/json/decoder.py", line 339, in decode obj, end = self.raw_decode(s, idx=_w(s, 0).end()) File "/soe/apblair/anaconda3/envs/single_cell/lib/python3.5/json/decoder.py", line 357, in raw_decode raise JSONDecodeError("Expecting value", s, err.value) from None json.decoder.JSONDecodeError: Expecting value: line 22 column 14 (char 549)

The ~/cell directory was created though and populated with the following directories/files:

cellbrowser.json.bak
discretMat.json
exprMatrix.json
meta.index
summary.html dataset.json
downloads.html
exprMatrix.tsv.gz
meta.tsv discretMat.bin
exprMatrix.bin
metaFields
methods.html

Thank you for the help!

[maximilianh]

Very strange, it's saying that the output file dataset.json is broken...

can you git pull and try again or alternatively send me your file dataset.json?

On Mon, Sep 10, 2018 at 10:23 PM, Andrew Blair notifications@github.com wrote:

After pulling the new commits, the sample data is running into a JSON decode error now.

cd sampleData/sample1/ ../../src/cbBuild -o ~/cells

Traceback (most recent call last): File "../../src/cbBuild", line 10, in cellbrowser.convertAndCopyCli() File "../../src/cbPyLib/cellbrowser.py", line 2246, in convertAndCopyCli convertAndCopy(confFnames, outDir, port) File "../../src/cbPyLib/cellbrowser.py", line 2220, in convertAndCopy convertDataset(inConf, outConf, datasetDir) File "../../src/cbPyLib/cellbrowser.py", line 2056, in convertDataset doMatrix = matrixOrSamplesHaveChanged(datasetDir, inMatrixFname, outMatrixFname, outConf) File "../../src/cbPyLib/cellbrowser.py", line 2001, in matrixOrSamplesHaveChanged lastConf = json.load(open(confName)) File "/soe/apblair/anaconda3/envs/single_cell/lib/python3.5/json/init.py", line 268, in load parse_constant=parse_constant, object_pairs_hook=object_pairs_hook, *kw) File "/soe/apblair/anaconda3/envs/single_cell/lib/python3.5/json/init*.py", line 319, in loads return _default_decoder.decode(s) File "/soe/apblair/anaconda3/envs/single_cell/lib/python3.5/json/decoder.py", line 339, in decode obj, end = self.raw_decode(s, idx=_w(s, 0).end()) File "/soe/apblair/anaconda3/envs/single_cell/lib/python3.5/json/decoder.py", line 357, in raw_decode raise JSONDecodeError("Expecting value", s, err.value) from None json.decoder.JSONDecodeError: Expecting value: line 22 column 14 (char 549)

The ~/cell directory was created though and populated with the following directories/files:

cellbrowser.json.bak discretMat.json exprMatrix.json meta.index summary.html dataset.json downloads.html exprMatrix.tsv.gz meta.tsv discretMat.bin exprMatrix.bin metaFields methods.html

Thank you for the help!

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/maximilianh/cellBrowser/issues/22#issuecomment-420047887, or mute the thread https://github.com/notifications/unsubscribe-auth/AAS-TZhxK8ZC52sbWgTnsaRlTMCXupadks5uZspagaJpZM4WfQjJ .

[apblair]

After the second git pull I am also receiving an error that the output file dataset.json is broken. I will email you the dataset.json file but also posting the contents here for future reference.

{ "desc": { "summary": "summary.html", "methods": "methods.html", "downloads": "downloads.html" }, "name": "sample", "shortLabel": "CellBrowser 100-genes demo", "radius": 5, "alpha": 0.3, "priority": 10, "tags": [ "smartseq2" ], "clusterField": "WGCNAcluster", "hubUrl": "http://cells.ucsc.edu/cortex-dev/hub/hub.txt", "showLabels": true, "ucscDb": "hg38", "fileVersions": { "inMeta": { "fname": "/projects/sysbio/users/apblair/single_cell/cellBrowser/sampleData/sample1/meta.tsv", "md5":

[maximilianh]

Yes, weird. This file is clearly truncated. The only explanation for this situation that I can see is that you did ctrl+c while the script was running...

I'll change it so this can't happen anymore.

And yes, there is still an error message, but the new error message suggested that you remove the file and restart the script. Could you try that?

On Tue, Sep 11, 2018 at 7:32 AM, Andrew Blair notifications@github.com wrote:

After the second git pull I am also receiving an error that the output file dataset.json is broken. I will email you the dataset.json file but also posting the contents here for future reference.

{ "desc": { "summary": "summary.html", "methods": "methods.html", "downloads": "downloads.html" }, "name": "sample", "shortLabel": "CellBrowser 100-genes demo", "radius": 5, "alpha": 0.3, "priority": 10, "tags": [ "smartseq2" ], "clusterField": "WGCNAcluster", "hubUrl": "http://cells.ucsc.edu/cortex-dev/hub/hub.txt", "showLabels": true, "ucscDb": "hg38", "fileVersions": { "inMeta": { "fname": "/projects/sysbio/users/apblair/single_cell/ cellBrowser/sampleData/sample1/meta.tsv", "md5":

— You are receiving this because you modified the open/close state. Reply to this email directly, view it on GitHub https://github.com/maximilianh/cellBrowser/issues/22#issuecomment-420151936, or mute the thread https://github.com/notifications/unsubscribe-auth/AAS-TViSQmbR-6UdrlOd3xcpnCuRqZKGks5uZ0rZgaJpZM4WfQjJ .

[apblair]

I very well might have hit ctrl+c at one point while running the script. I removed the dataset.json from ~/cells and did another git pull but I'm running into a JSON serializable error now:

Traceback (most recent call last): File "../../src/cbBuild", line 10, in cellbrowser.convertAndCopyCli() File "../../src/cbPyLib/cellbrowser.py", line 2252, in convertAndCopyCli convertAndCopy(confFnames, outDir, port) File "../../src/cbPyLib/cellbrowser.py", line 2226, in convertAndCopy convertDataset(inConf, outConf, datasetDir) File "../../src/cbPyLib/cellbrowser.py", line 2066, in convertDataset writeConfig(inConf, outConf, datasetDir) File "../../src/cbPyLib/cellbrowser.py", line 2183, in writeConfig json.dump(outConf, descJsonFh, indent=2) File "/soe/apblair/anaconda3/lib/python3.6/json/init.py", line 179, in dump for chunk in iterable: File "/soe/apblair/anaconda3/lib/python3.6/json/encoder.py", line 430, in _iterencode yield from _iterencode_dict(o, _current_indent_level) File "/soe/apblair/anaconda3/lib/python3.6/json/encoder.py", line 404, in _iterencode_dict yield from chunks File "/soe/apblair/anaconda3/lib/python3.6/json/encoder.py", line 404, in _iterencode_dict yield from chunks File "/soe/apblair/anaconda3/lib/python3.6/json/encoder.py", line 404, in _iterencode_dict yield from chunks File "/soe/apblair/anaconda3/lib/python3.6/json/encoder.py", line 437, in _iterencode o = _default(o) File "/soe/apblair/anaconda3/lib/python3.6/json/encoder.py", line 180, in default o.class.name) TypeError: Object of type 'bytes' is not JSON serializable

After the git pull I also tried deleting the ~/cells directory and then running the sample dataset, but I still run into the broken dataset.json message.

[maximilianh]

Oh! Now I understand! You're running this on python3! Sorry! (btw: what is your platform? I'm surprised that your platform uses python3 as the default python...)

Once I figured out that it's on python 3 (thanks for sending the full error message this time, it was very easy to fix. Can you try again?

[apblair]

I'm working on the stuart lab server (tap), which I believe is managed by BSOE, and the platform is CentOS 7.5.1804 release. I installed anaconda and upgraded to python 3.5. But it looks like it works now https://users.soe.ucsc.edu/~apblair/

Thanks again for your help :+1: