search

GMOD / Apollo3

JBrowse 2 plugin for editing annotations on an Apollo server
Apache License 2.0
6 stars 4 forks source link

Add more CLI functionality #379

Closed dariober closed 2 months ago

dariober commented 3 months ago

Several additions to the CLI

garrettjstevens commented 3 months ago

Hi @dariober, I couldn't ever get the FormData to stream properly, so I tried another approach, creating a new endpoint to stream files to without them being in FormData. The branch is here: https://github.com/GMOD/Apollo3/tree/cli-misc-gjs

Could you try out this branch and see if it works on your end?

dariober commented 3 months ago

@garrettjstevens Thanks - but still no luck I'm afraid! In the following commands apollo is an alias for alias apollo='yarn --cwd /home/dario/Apollo/Apollo3/packages/apollo-cli dev'. 

git branch --show-current
cli-misc-gjs

git rev-parse HEAD
329e82fa0bab29dc788fcd31bf6cd3c13b06c2a3

Loading a small fasta file: I executed the same command 3 times. The first two succeeded the third failed (you can see the time of execution on the left):

12:13:27 apollo-cli$ apollo assembly add-fasta -i test_data/tiny.fasta -a test -f 
progress [========================================] 100% | ETA: 0s | 1247/1247

12:14:20 apollo-cli$ apollo assembly add-fasta -i test_data/tiny.fasta -a test -f 
progress [========================================] 100% | ETA: 0s | 1247/1247

12:14:34 apollo-cli$ apollo assembly add-fasta -i test_data/tiny.fasta -a test -f 
progress [----------------------------------------] 0% | ETA: 0s | 0/1247TypeError: fetch failed
    at fetch (/home/dario/Apollo/Apollo3/.yarn/cache/undici-npm-6.7.0-72b23a7b9b-bc03abd66d.zip/node_modules/undici/index.js:105:13)
    at async uploadFile (file:///home/dario/Apollo/Apollo3/packages/apollo-cli/src/utils.ts:4:2639)
    at async Get.run (file:///home/dario/Apollo/Apollo3/packages/apollo-cli/src/commands/assembly/add-fasta.ts:1:1972)
    at async Get._run (/home/dario/Apollo/Apollo3/.yarn/cache/@oclif-core-npm-3.18.2-253d4df2a5-1f04aefcf8.zip/node_modules/@oclif/core/lib/command.js:304:22)
    at async Config.runCommand (/home/dario/Apollo/Apollo3/.yarn/cache/@oclif-core-npm-3.18.2-253d4df2a5-1f04aefcf8.zip/node_modules/@oclif/core/lib/config/config.js:417:25)
    at async run (/home/dario/Apollo/Apollo3/.yarn/cache/@oclif-core-npm-3.18.2-253d4df2a5-1f04aefcf8.zip/node_modules/@oclif/core/lib/main.js:85:16)
    at async main (file:///home/dario/Apollo/Apollo3/packages/apollo-cli/bin/dev.js:6:3)
    at async file:///home/dario/Apollo/Apollo3/packages/apollo-cli/bin/dev.js:9:1 {
  [cause]: RequestContentLengthMismatchError: Request body length does not match content-length header
      at AsyncWriter.end (/home/dario/Apollo/Apollo3/.yarn/cache/undici-npm-6.7.0-72b23a7b9b-bc03abd66d.zip/node_modules/undici/lib/dispatcher/client-h1.js:1311:15)
      at writeIterable (/home/dario/Apollo/Apollo3/.yarn/cache/undici-npm-6.7.0-72b23a7b9b-bc03abd66d.zip/node_modules/undici/lib/dispatcher/client-h1.js:1194:12)
      at processTicksAndRejections (node:internal/process/task_queues:95:5) {
    code: 'UND_ERR_REQ_CONTENT_LENGTH_MISMATCH'
  }
}
progress [========================================] 100% | ETA: 0s | 1247/1247
TypeError: fetch failed
    at fetch (/home/dario/Apollo/Apollo3/.yarn/cache/undici-npm-6.7.0-72b23a7b9b-bc03abd66d.zip/node_modules/undici/index.js:105:13)
    at async uploadFile (file:///home/dario/Apollo/Apollo3/packages/apollo-cli/src/utils.ts:4:2639)
    at async Get.run (file:///home/dario/Apollo/Apollo3/packages/apollo-cli/src/commands/assembly/add-fasta.ts:1:1972)
    at async Get._run (/home/dario/Apollo/Apollo3/.yarn/cache/@oclif-core-npm-3.18.2-253d4df2a5-1f04aefcf8.zip/node_modules/@oclif/core/lib/command.js:304:22)
    at async Config.runCommand (/home/dario/Apollo/Apollo3/.yarn/cache/@oclif-core-npm-3.18.2-253d4df2a5-1f04aefcf8.zip/node_modules/@oclif/core/lib/config/config.js:417:25)
    at async run (/home/dario/Apollo/Apollo3/.yarn/cache/@oclif-core-npm-3.18.2-253d4df2a5-1f04aefcf8.zip/node_modules/@oclif/core/lib/main.js:85:16)
    at async main (file:///home/dario/Apollo/Apollo3/packages/apollo-cli/bin/dev.js:6:3)
    at async file:///home/dario/Apollo/Apollo3/packages/apollo-cli/bin/dev.js:9:1

This is the server log for the failing command:

[Nest] 1334  - 04/09/2024, 11:16:21 AM   DEBUG [AuthorizationValidation] Required role is 'readOnly'
[Nest] 1334  - 04/09/2024, 11:16:21 AM   DEBUG [AuthorizationValidation] Calling class 'AssembliesController' and endpoint 'findAll'
[Nest] 1334  - 04/09/2024, 11:16:21 AM   DEBUG [AuthorizationValidation] Required role is 'admin'
[Nest] 1334  - 04/09/2024, 11:16:21 AM   DEBUG [AuthorizationValidation] Calling class 'FilesController' and endpoint 'streamFile'
[Nest] 1334  - 04/09/2024, 11:16:21 AM     LOG [FilesService] Starting file upload: "tiny.fasta"
[Nest] 1334  - 04/09/2024, 11:16:21 AM   DEBUG [FilesService] Uploading to temporary file "test/uploaded/upload-tmp-ETvOLZ/tiny.fasta.gz"
[Nest] 1334  - 04/09/2024, 11:16:21 AM   ERROR [GlobalExceptionsFilter] Error: aborted
[Nest] 1334  - 04/09/2024, 11:16:21 AM   ERROR [ExceptionsHandler] aborted
Error: aborted
    at connResetException (node:internal/errors:717:14)
    at abortIncoming (node:_http_server:754:17)
    at socketOnClose (node:_http_server:748:3)
    at Socket.emit (node:events:525:35)
    at TCP.<anonymous> (node:net:322:12)

For a larger input, the command gets stuck at:

wc test_data/tmp.fa
 10001  10001 460006 test_data/tmp.fa

apollo assembly add-fasta -i test_data/tmp.fa -a test -f 
progress [========================================] 100% | ETA: 0s | 460006/460006

With server log:

[Nest] 1130  - 04/09/2024, 11:19:52 AM   DEBUG [AuthorizationValidation] Required role is 'readOnly'
[Nest] 1130  - 04/09/2024, 11:19:52 AM   DEBUG [AuthorizationValidation] Calling class 'AssembliesController' and endpoint 'findAll'
[Nest] 1130  - 04/09/2024, 11:19:52 AM   DEBUG [AuthorizationValidation] Required role is 'admin'
[Nest] 1130  - 04/09/2024, 11:19:52 AM   DEBUG [AuthorizationValidation] Calling class 'FilesController' and endpoint 'streamFile'
[Nest] 1130  - 04/09/2024, 11:19:52 AM     LOG [FilesService] Starting file upload: "tmp.fa"
[Nest] 1130  - 04/09/2024, 11:19:52 AM   DEBUG [FilesService] Uploading to temporary file "test/uploaded/upload-tmp-fCsjrw/tmp.fa.gz"
[Nest] 1130  - 04/09/2024, 11:19:52 AM   DEBUG [FilesService] File upload progress: 14.08%