BioCompute Objects no longer exportable from invocation

[HadleyKing]

It appears that BioCompute Objects no longer exportable from invocation

The button on the invocation page is gone and the API download is causing a server error.

[davelopez]

Hi @HadleyKing, the export button is not present in usegalaxy.org because the server instance doesn't have the Celery tasks system enabled. This feature will be available on servers with Celery enabled and in usegalaxy.org once a known issue with this system is resolved. We can add back the old BCO download link meanwhile, sorry about that.

As for the Internal Server Error in api/invocations/{invocation_id}/biocompute/download I can't reproduce it in a local instance but I get the same 500 error on usegalaxy.org, so it would be nice if someone with access to usegalaxy.org logs can give us a hint of what is failing here...

[martenson]

@davelopez

Feb 28 11:50:33 galaxy-web-05 galaxyctl[1813]: uvicorn.access INFO 2023-02-28 11:50:33,365 [pN:main.4,p:9101,tN:MainThread] 95.208.248.166:0 - "GET /api/invocations/<ID>/biocompute/download HTTP/1.0" 500
Feb 28 11:50:33 galaxy-web-05 galaxyctl[1813]: [2023-02-28 11:50:33 -0600] [9101] [ERROR] Exception in ASGI application
Feb 28 11:50:33 galaxy-web-05 galaxyctl[1813]: Traceback (most recent call last):
Feb 28 11:50:33 galaxy-web-05 galaxyctl[1813]: File "/cvmfs/main.galaxyproject.org/venv/lib/python3.8/site-packages/anyio/streams/memory.py", line 94, in receive
Feb 28 11:50:33 galaxy-web-05 galaxyctl[1813]: return self.receive_nowait()
Feb 28 11:50:33 galaxy-web-05 galaxyctl[1813]: File "/cvmfs/main.galaxyproject.org/venv/lib/python3.8/site-packages/anyio/streams/memory.py", line 89, in receive_nowait
Feb 28 11:50:33 galaxy-web-05 galaxyctl[1813]: raise WouldBlock
Feb 28 11:50:33 galaxy-web-05 galaxyctl[1813]: anyio.WouldBlock
Feb 28 11:50:33 galaxy-web-05 galaxyctl[1813]: During handling of the above exception, another exception occurred:
Feb 28 11:50:33 galaxy-web-05 galaxyctl[1813]: Traceback (most recent call last):
Feb 28 11:50:33 galaxy-web-05 galaxyctl[1813]: File "/cvmfs/main.galaxyproject.org/venv/lib/python3.8/site-packages/starlette/middleware/base.py", line 78, in call_next
Feb 28 11:50:33 galaxy-web-05 galaxyctl[1813]: message = await recv_stream.receive()
Feb 28 11:50:33 galaxy-web-05 galaxyctl[1813]: File "/cvmfs/main.galaxyproject.org/venv/lib/python3.8/site-packages/anyio/streams/memory.py", line 114, in receive
Feb 28 11:50:33 galaxy-web-05 galaxyctl[1813]: raise EndOfStream
Feb 28 11:50:33 galaxy-web-05 galaxyctl[1813]: anyio.EndOfStream
Feb 28 11:50:33 galaxy-web-05 galaxyctl[1813]: During handling of the above exception, another exception occurred:
Feb 28 11:50:33 galaxy-web-05 galaxyctl[1813]: Traceback (most recent call last):
Feb 28 11:50:33 galaxy-web-05 galaxyctl[1813]: File "/cvmfs/main.galaxyproject.org/venv/lib/python3.8/site-packages/uvicorn/protocols/http/h11_impl.py", line 407, in run_asgi
Feb 28 11:50:33 galaxy-web-05 galaxyctl[1813]: result = await app(  # type: ignore[func-returns-value]
Feb 28 11:50:33 galaxy-web-05 galaxyctl[1813]: File "/cvmfs/main.galaxyproject.org/venv/lib/python3.8/site-packages/uvicorn/middleware/proxy_headers.py", line 78, in __call__
Feb 28 11:50:33 galaxy-web-05 galaxyctl[1813]: return await self.app(scope, receive, send)

[HadleyKing]

@davelopez That would be great. We have a couple of collaborators who are using this feature.

@agbiotec and @tiwa1154 keep an eye on this issue.

[davelopez]

Thank you @martenson for the log! Unfortunately, It doesn't tell me that there is a direct issue with the Galaxy code itself that I can fix.

@mvdbeek, do you think this can be a deployment issue in usegalaxy.org affecting StreamingResponse? This endpoint and the one in #15390 for collection downloads use StreamingResponse so I also tried https://usegalaxy.org/api/pages/{id}.pdf to see if there is a pattern and looks like it fails with a 500 too...

[mvdbeek]

Don't we need Celery for both of these ?

[davelopez]

As far as I know, none of these 3 endpoints use Celery:

https://github.com/galaxyproject/galaxy/blob/abe68f72776c91f59bb9c2f16a659a7b2fd8661c/lib/galaxy/webapps/galaxy/api/workflows.py#L1458

https://github.com/galaxyproject/galaxy/blob/abe68f72776c91f59bb9c2f16a659a7b2fd8661c/lib/galaxy/webapps/galaxy/api/pages.py#L144

https://github.com/galaxyproject/galaxy/blob/abe68f72776c91f59bb9c2f16a659a7b2fd8661c/lib/galaxy/webapps/galaxy/api/history_contents.py#L491

The ones that use Celery are the prepare_download alternatives.

[mvdbeek]

https://usegalaxy.org/api/pages/{id}.pdf

that's a 501, right ? https://github.com/mvdbeek/galaxy/blob/9b26f9eb4d6222f14cfe91e08bb11baa0f0b0f29/lib/galaxy/webapps/galaxy/api/pages.py#L152

[davelopez]

I just tried with a simple page: https://usegalaxy.org/published/page?id=49e017f8b0fb62ec

And unfortunately, I get a proper 500 with https://usegalaxy.org/api/pages/49e017f8b0fb62ec.pdf

[mvdbeek]

https://usegalaxy.org/api/pages/49e017f8b0fb62ec.pdf

that one is https://sentry.galaxyproject.org/share/issue/bb794e223b5a4007a517153691284110/

WouldBlock: null
  File "anyio/streams/memory.py", line 94, in receive
    return self.receive_nowait()
  File "anyio/streams/memory.py", line 89, in receive_nowait
    raise WouldBlock
EndOfStream: null
  File "starlette/middleware/base.py", line 78, in call_next
    message = await recv_stream.receive()
  File "anyio/streams/memory.py", line 114, in receive
    raise EndOfStream
AttributeError: function/symbol 'pango_context_set_round_glyph_positions' not found in library 'libpango-1.0.so.0': /lib64/libpango-1.0.so.0: undefined symbol: pango_context_set_round_glyph_positions
  File "starlette/applications.py", line 118, in __call__
    await self.middleware_stack(scope, receive, send)
  File "starlette/middleware/errors.py", line 184, in __call__
    raise exc
  File "starlette/middleware/errors.py", line 162, in __call__
    await self.app(scope, receive, _send)
  File "starlette_context/middleware/raw_middleware.py", line 93, in __call__
    await self.app(scope, receive, send_wrapper)
  File "starlette/middleware/base.py", line 108, in __call__
    response = await self.dispatch_func(request, call_next)
  File "galaxy/webapps/galaxy/fast_app.py", line 103, in add_x_frame_options
    response = await call_next(request)
  File "starlette/middleware/base.py", line 84, in call_next
    raise app_exc
  File "starlette/middleware/base.py", line 70, in coro
    await self.app(scope, receive_or_disconnect, send_no_error)
  File "starlette/middleware/exceptions.py", line 79, in __call__
    raise exc
  File "starlette/middleware/exceptions.py", line 68, in __call__
    await self.app(scope, receive, sender)
  File "fastapi/middleware/asyncexitstack.py", line 21, in __call__
    raise e
  File "fastapi/middleware/asyncexitstack.py", line 18, in __call__
    await self.app(scope, receive, send)
  File "starlette/routing.py", line 706, in __call__
    await route.handle(scope, receive, send)
  File "starlette/routing.py", line 276, in handle
    await self.app(scope, receive, send)
  File "starlette/routing.py", line 66, in app
    response = await func(request)
  File "fastapi/routing.py", line 237, in app
    raw_response = await run_endpoint_function(
  File "fastapi/routing.py", line 163, in run_endpoint_function
    return await dependant.call(**values)
  File "galaxy/webapps/galaxy/api/pages.py", line 164, in show_pdf
    pdf_bytes = self.service.show_pdf(trans, id)
  File "galaxy/webapps/galaxy/services/pages.py", line 124, in show_pdf
    return internal_galaxy_markdown_to_pdf(trans, internal_galaxy_markdown, PdfDocumentType.page)
  File "galaxy/managers/markdown_util.py", line 620, in internal_galaxy_markdown_to_pdf
    return to_branded_pdf(basic_markdown, document_type, config)
  File "galaxy/managers/markdown_util.py", line 651, in to_branded_pdf
    return to_pdf_raw(branded_markdown, css_paths=css_paths)
  File "galaxy/managers/markdown_util.py", line 601, in to_pdf_raw
    return html.write_pdf(stylesheets=stylesheets)
  File "__init__.py", line 198, in write_pdf
    self.render(
  File "__init__.py", line 143, in render
    return Document._render(
  File "weasyprint/document.py", line 219, in _render
    [Page(page_box) for page_box in page_boxes],
  File "weasyprint/document.py", line 219, in <listcomp>
    [Page(page_box) for page_box in page_boxes],
  File "weasyprint/layout/__init__.py", line 128, in layout_document
    pages = list(make_all_pages(context, root_box, html, pages))
  File "weasyprint/layout/page.py", line 852, in make_all_pages
    page, resume_at = remake_page(i, context, root_box, html)
  File "weasyprint/layout/page.py", line 788, in remake_page
    page, resume_at, next_page = make_page(
  File "weasyprint/layout/page.py", line 586, in make_page
    root_box, resume_at, next_page, _, _, _ = block_level_layout(
  File "weasyprint/layout/block.py", line 59, in block_level_layout
    return block_level_layout_switch(
  File "weasyprint/layout/block.py", line 75, in block_level_layout_switch
    return block_box_layout(
  File "weasyprint/layout/block.py", line 120, in block_box_layout
    result = block_container_layout(
  File "weasyprint/layout/block.py", line 686, in block_container_layout
    next_page, new_children, new_max_lines) = _in_flow_layout(
  File "weasyprint/layout/block.py", line 482, in _in_flow_layout
    collapsing_through, max_lines) = block_level_layout(
  File "weasyprint/layout/block.py", line 59, in block_level_layout
    return block_level_layout_switch(
  File "weasyprint/layout/block.py", line 75, in block_level_layout_switch
    return block_box_layout(
  File "weasyprint/layout/block.py", line 120, in block_box_layout
    result = block_container_layout(
  File "weasyprint/layout/block.py", line 686, in block_container_layout
    next_page, new_children, new_max_lines) = _in_flow_layout(
  File "weasyprint/layout/block.py", line 482, in _in_flow_layout
    collapsing_through, max_lines) = block_level_layout(
  File "weasyprint/layout/block.py", line 59, in block_level_layout
    return block_level_layout_switch(
  File "weasyprint/layout/block.py", line 75, in block_level_layout_switch
    return block_box_layout(
  File "weasyprint/layout/block.py", line 120, in block_box_layout
    result = block_container_layout(
  File "weasyprint/layout/block.py", line 675, in block_container_layout
    new_footnotes, max_lines) = _linebox_layout(
  File "weasyprint/layout/block.py", line 319, in _linebox_layout
    for i, (line, resume_at) in enumerate(lines_iterator):
  File "weasyprint/layout/inline.py", line 38, in iter_line_boxes
    line, resume_at = get_next_linebox(
  File "weasyprint/layout/inline.py", line 95, in get_next_linebox
    last_letter, float_width) = split_inline_box(
  File "weasyprint/layout/inline.py", line 706, in split_inline_box
    split_inline_level(
  File "weasyprint/layout/inline.py", line 469, in split_inline_level
    new_box, skip, preserved_line_break = split_text_box(
  File "weasyprint/layout/inline.py", line 877, in split_text_box
    layout, length, resume_index, width, height, baseline = split_first_line(
  File "weasyprint/text/line_break.py", line 307, in split_first_line
    layout = create_layout(
  File "weasyprint/text/line_break.py", line 256, in create_layout
    layout = Layout(
  File "weasyprint/text/line_break.py", line 64, in __init__
    self.setup(context, font_size, style)
  File "weasyprint/text/line_break.py", line 80, in setup
    pango.pango_context_set_round_glyph_positions(pango_context, False)
  File "cffi/api.py", line 912, in __getattr__
    make_accessor(name)
  File "cffi/api.py", line 908, in make_accessor
    accessors[name](name)
  File "cffi/api.py", line 838, in accessor_function
    value = backendlib.load_function(BType, name)

which I guess means we need to update pango

[mvdbeek]

https://usegalaxy.org/api/invocations/56e22c436310b00c/biocompute/download works fine for me.

For @HadleyKing's invocation this is https://sentry.galaxyproject.org/share/issue/f73c4e49b99d4c2fbb81cfbbc7de3311/:

WouldBlock: null
  File "anyio/streams/memory.py", line 94, in receive
    return self.receive_nowait()
  File "anyio/streams/memory.py", line 89, in receive_nowait
    raise WouldBlock
EndOfStream: null
  File "starlette/middleware/base.py", line 78, in call_next
    message = await recv_stream.receive()
  File "anyio/streams/memory.py", line 114, in receive
    raise EndOfStream
ValidationError: 1 validation error for Uri
uri
  URL invalid, extra characters found after valid URL: ' beginning1' (type=value_error.url.extra; extra= beginning1)
  File "starlette/applications.py", line 118, in __call__
    await self.middleware_stack(scope, receive, send)
  File "starlette/middleware/errors.py", line 184, in __call__
    raise exc
  File "starlette/middleware/errors.py", line 162, in __call__
    await self.app(scope, receive, _send)
  File "starlette_context/middleware/raw_middleware.py", line 93, in __call__
    await self.app(scope, receive, send_wrapper)
  File "starlette/middleware/base.py", line 108, in __call__
    response = await self.dispatch_func(request, call_next)
  File "galaxy/webapps/galaxy/fast_app.py", line 103, in add_x_frame_options
    response = await call_next(request)
  File "starlette/middleware/base.py", line 84, in call_next
    raise app_exc
  File "starlette/middleware/base.py", line 70, in coro
    await self.app(scope, receive_or_disconnect, send_no_error)
  File "starlette/middleware/exceptions.py", line 79, in __call__
    raise exc
  File "starlette/middleware/exceptions.py", line 68, in __call__
    await self.app(scope, receive, sender)
  File "fastapi/middleware/asyncexitstack.py", line 21, in __call__
    raise e
  File "fastapi/middleware/asyncexitstack.py", line 18, in __call__
    await self.app(scope, receive, send)
  File "starlette/routing.py", line 706, in __call__
    await route.handle(scope, receive, send)
  File "starlette/routing.py", line 276, in handle
    await self.app(scope, receive, send)
  File "starlette/routing.py", line 66, in app
    response = await func(request)
  File "fastapi/routing.py", line 237, in app
    raw_response = await run_endpoint_function(
  File "fastapi/routing.py", line 165, in run_endpoint_function
    return await run_in_threadpool(dependant.call, **values)
  File "starlette/concurrency.py", line 41, in run_in_threadpool
    return await anyio.to_thread.run_sync(func, *args)
  File "anyio/to_thread.py", line 31, in run_sync
    return await get_asynclib().run_sync_in_worker_thread(
  File "anyio/_backends/_asyncio.py", line 937, in run_sync_in_worker_thread
    return await future
  File "anyio/_backends/_asyncio.py", line 867, in run
    result = context.run(func, *args)
  File "galaxy/webapps/galaxy/api/workflows.py", line 1487, in download_invocation_bco
    bco = self._deprecated_generate_bco(trans, invocation_id, merge_history_metadata)
  File "galaxy/webapps/galaxy/api/workflows.py", line 1506, in _deprecated_generate_bco
    return self.invocations_service.deprecated_generate_invocation_bco(trans, invocation_id, export_options)
  File "galaxy/webapps/galaxy/services/invocations.py", line 235, in deprecated_generate_invocation_bco
    export_target.seek(0)
  File "galaxy/model/store/__init__.py", line 2364, in __exit__
    self._finalize()
  File "galaxy/model/store/__init__.py", line 2509, in _finalize
    core_biocompute_object, object_id = self._core_biocompute_object_and_object_id()
  File "galaxy/model/store/__init__.py", line 2555, in _core_biocompute_object_and_object_id
    software_prerequisite_tracker.register_step(workflow_step)
  File "galaxy/model/store/_bco_convert_utils.py", line 40, in register_step
    uri=ExecutionDomainUri(uri=uri, access_time=access_time),
  File "pydantic/main.py", line 342, in pydantic.main.BaseModel.__init__
    raise validation_error