Docker - Hosting an interactive viewer issue

[CRichard21]

Hello,

I am trying to run blobtoolkit docker container locally but I have some issue with the Viewer part.

Here what I ran:

docker pull genomehubs/blobtoolkit

docker run -it --rm --name btk \
           -u $UID:$GROUPS \
           -v $HOME/Blobtools/datasets/:/blobtoolkit/datasets \
           -v $HOME/Blobtools/data/:/blobtoolkit/data \
           genomehubs/blobtoolkit:latest \
           ./blobtools2/blobtools create \
           --fasta data/scaffolds.fasta \
           --busco data/busco/run_bacteria_odb10/full_table.tsv \
           --cov data/assembly.reads.bam \
           datasets/BACTWG1

docker run -d --rm --name btk \
           -v $HOME/Blobtools/datasets/:/blobtoolkit/datasets \
           -p 8000:8000 -p 8080:8080 \
           -e VIEWER=true \
           genomehubs/blobtoolkit:latest

All ran well and the last command give me the output 861791b489f6ae0d12b5d4c8fbd481af53fe2c5f2fba378d339c7d43ec125435.

However, when I open http://localhost:8080/view/all in a web browser (Google Chrome), i only get an empty page with the message: "This page isn't working localhost didn't send any data. ERR_EMPTY_RESPONSE"

When I take a look at the docker instance in the docker desktop, I can find the log file:

> blobtoolkit-viewer@1.2.1 start /blobtoolkit/viewer
> nodemon src/server/app.js

[nodemon] 2.0.2
[nodemon] to restart at any time, enter `rs`
[nodemon] watching dir(s): *.*
[nodemon] watching extensions: js,mjs,json
[nodemon] starting `node src/server/app.js`
BlobToolKit RESTful API server started on http port: 8000

> blobtoolkit-viewer@1.2.1 client /blobtoolkit/viewer
> webpack-dev-server --mode development --content-base src/client

fatal: not a git repository (or any of the parent directories): .git
child_process.js:669
    throw err;
    ^

Error: Command failed: git describe --always
fatal: not a git repository (or any of the parent directories): .git

    at checkExecSyncError (child_process.js:629:11)
    at execSync (child_process.js:666:13)
    at module.exports (/blobtoolkit/viewer/node_modules/git-revision-webpack-plugin/lib/helpers/run-git-command.js:25:34)
    at GitRevisionPlugin.version (/blobtoolkit/viewer/node_modules/git-revision-webpack-plugin/lib/index.js:65:10)
    at Object.<anonymous> (/blobtoolkit/viewer/webpack.config.js:81:53)
    at Module._compile (internal/modules/cjs/loader.js:778:30)
    at Object.Module._extensions..js (internal/modules/cjs/loader.js:789:10)
    at Module.load (internal/modules/cjs/loader.js:653:32)
    at tryModuleLoad (internal/modules/cjs/loader.js:593:12)
    at Function.Module._load (internal/modules/cjs/loader.js:585:3)
    at Module.require (internal/modules/cjs/loader.js:692:17)
    at require (internal/modules/cjs/helpers.js:25:18)
    at WEBPACK_OPTIONS (/blobtoolkit/viewer/node_modules/webpack-cli/bin/utils/convert-argv.js:114:13)
    at requireConfig (/blobtoolkit/viewer/node_modules/webpack-cli/bin/utils/convert-argv.js:116:6)
    at /blobtoolkit/viewer/node_modules/webpack-cli/bin/utils/convert-argv.js:123:17
    at Array.forEach (<anonymous>)
    at module.exports (/blobtoolkit/viewer/node_modules/webpack-cli/bin/utils/convert-argv.js:121:15)
    at Object.<anonymous> (/blobtoolkit/viewer/node_modules/webpack-dev-server/bin/webpack-dev-server.js:84:40)
    at Module._compile (internal/modules/cjs/loader.js:778:30)
    at Object.Module._extensions..js (internal/modules/cjs/loader.js:789:10)
    at Module.load (internal/modules/cjs/loader.js:653:32)
    at tryModuleLoad (internal/modules/cjs/loader.js:593:12)
npm ERR! code ELIFECYCLE
npm ERR! errno 1
npm ERR! blobtoolkit-viewer@1.2.1 client: `webpack-dev-server --mode development --content-base src/client`
npm ERR! Exit status 1

npm ERR! 
npm ERR! Failed at the blobtoolkit-viewer@1.2.1 client script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.

npm ERR! A complete log of this run can be found in:
npm ERR!     /home/blobtoolkit/.npm/_logs/2021-04-11T23_43_21_245Z-debug.log

Can you please help me to fix this issue ?

For information, I am working on Mac OS Big Sur v11.2.3.

Best,

Corentin

[CRichard21]

Hello again,

I saw it was related to the issue in the viewer page here but I was not with the use of docker. I did not expect this error with the container.

How can I solve that?

Best,

Corentin

[rjchallis]

Hi Corentin

This is a problem with the container image - the .git directories were deleted which caused an error when the viewer tried to execute a git command to check the version. I had updated it a few months ago, but the "latest" tag was still pointing to an older version. I've just updated this in docker hub so if you run docker pull genomehubs/blobtoolkit:latest, you should get a more recent version with the .git directories preserved.

Hopefully the newer version will work as expected

Rich

[CRichard21]

Hello Richard,

Thank you! I succeed in accessed to the Viewer. I can now see snail plot. But two issues/questions remain for me:

1- As you can see on the snail plot attached, BUSCO plot show a lot of missing. However, the short_summary BUSCO file give me:

        C:100.0%[S:20.2%,D:79.8%],F:0.0%,M:0.0%,n:124      
        124 Complete BUSCOs (C)            
        25  Complete and single-copy BUSCOs (S)    
        99  Complete and duplicated BUSCOs (D)     
        0   Fragmented BUSCOs (F)              
        0   Missing BUSCOs (M)             
        124 Total BUSCO groups searched

and an empty missing BUSCO list file. It looks like the BUSCO plot is done on the number of genome from the lineage (4085 for bacteria_odb10) and not the number of BUSCO. Am I right ? Is this result expected ? I used BUSCO v5.1.2_cv1.

2- I can see snail plot but not cumulative plot.

Thus, I ran blastn v2.11.0 command to get a hits file as follow:

blastn -db ncbi_nt/nt \
       -query data/scaffolds.fasta \
       -outfmt "6 qseqid staxids bitscore std" \
       -max_target_seqs 10 \
       -max_hsps 1 \
       -evalue 1e-25 \
       -out blastn_results.out

and then, I create my dataset:

docker run -it --rm --name btk \
           -u $UID:$GROUPS \
           -v $HOME/Blobtools/datasets/:/blobtoolkit/datasets \
           -v $HOME/Blobtools/data/:/blobtoolkit/data \
           genomehubs/blobtoolkit:latest \
           ./blobtools2/blobtools create \
           --fasta data/scaffolds.fasta \
           --busco data/busco/run_bacteria_odb10/full_table.tsv \
           --cov data/assembly.reads.bam \
           --hits data/blastn_results.out \
           --taxdump databases/ncbi_taxdump \
           datasets/BACTWG1

However, when I look at the Viewer and the cumulative plot, the web page stay white. When I want to generate the plot on the command line, it is stucked on these lines:

Loading http://localhost:8080/view/dataset/BACTWG1/cumulative?staticThreshold=Infinity&nohitThreshold=Infinity&plotGraphics=svg
Fetching BACTWG1.cumulative.png
waiting for element cumulative_save_png

How can I solve this issue ?

Thanks a lot for your help and the amazing job with Blobtoolkit!

Best,

Corentin

[rjchallis]

Glad to hear the viewer is working for you now!

1. Yes, it looks like the number of genomes is being used instead of the number of genes. I had hacked my BUSCO parser to get around BUSCO occasionally reporting these 2 values the wrong way around, but in this case my test gets it wrong and flips the values when it shouldn't. To get this working with the current version you can just swap the values for number of genes and number of genomes in the full_table.tsv header. I'll fix this properly in the next release.
2. I'm not sure what is happening here - could you show me the listing for the BlobDir directory so I can check that the expected files have been created. It would also be useful to know if there are any error messages in the javascript console when the browser shows a white screen. In Chrome this can be found under View > Developer > Javascript Console. In other browsers it may be under a Develop menu or Tools > Web Developer.

[CRichard21]

Hello Richard,

Here what I can find in the datasets/BACTWG1 directory:

assembly.reads_cov.json       gc.json       meta.json
assembly.reads_read_cov.json  identifiers.json  ncount.json
bacteria_odb10_busco.json     length.json

I does not seem to have a file related to blastn output, does it ? I also attach what I found in the Javascript Console. When I click on "cumulative", two errors occured.

Best,

Corentin

[rjchallis]

ok, so it seems that importing the blast results didn't work. If you re-run the blobtools command with a --replace flag (or use the blobtools replace alias) it will overwrite the existing BlobDir and create a new on from scratch, which will hopefully solve this. Alternatively, you could use

blobtools add --hits data/blastn_results.out \
    --taxdump databases/ncbi_taxdump \
    datasets/BACTWG1

(leaving out the fasta and coverage files) to try just adding hits to the existing BlobDir.

[CRichard21]

Hello Richard,

I tryed the option you gave me with blobtools add. However, it still does not work. I also tryed to draw the cumulative plot with the Strongyloides venezuelensis fasta and hits files you gave for testing. I ran the following command :

docker run -it --rm --name btk \
           -u $UID:$GROUPS \
           -v $HOME/Blobtools/datasets/:/blobtoolkit/datasets \
           -v $HOME/Blobtools/data/:/blobtoolkit/data \
           genomehubs/blobtoolkit:latest \
           ./blobtools2/blobtools create \
           --fasta data/ASSEMBLY.fasta \
           datasets/ASSEMBLY

 docker run -it --rm --name btk \
           -u $UID:$GROUPS \
           -v $HOME/Blobtools/datasets/:/blobtoolkit/datasets \
           -v $HOME/Blobtools/data/:/blobtoolkit/data \
           genomehubs/blobtoolkit:latest \
           ./blobtools2/blobtools add \
          --hits data/ASSEMBLY.out \
          --taxdump databases/ncbi_taxdump \
           datasets/ASSEMBLY  

I succeed to get the snail plot but still not the cumulative plot so I do not think it comes from the file. There is still no more created file in the datasets/ASSEMBLY folder.

Is there a way we can solve this ? Is there something I am doing wrong ? Did I have to download some more ? I can send you my files if needed.

Best,

Corentin

[github-actions[bot]]

This issue has being marked as dormant because blobtoolkit-docker is being archived. The code is now part of the main BlobToolKit repository.

If you feel the issue has not been resolved, please follow the updated BlobToolKit installation instructions to first confirm that you are using the latest version, and then open a new issue at the main BlobToolKit repository if necessary.

This issue will be automatically closed in 7 days.

[github-actions[bot]]

This issue was closed because it has been inactive for 7 days since being marked as dormant.