Support serving a Site of Apps from a configuration file

[MarcSkovMadsen]

As soon as your collection of apps and scripts starts to grow, questions starts coming in

• How do my users easily navigate when using the site?
• Where do I find the code, documentation, ....?
• Who developed or owns this particular app?
• How can I easily manage which files to serve and where?
• How can I customize the url of a specific app?. I.e. not use /spread_app but /spread-analysis or even /trading/spread-analysis for my file spread_app.py.

In my experience a configuration file is needed. Currently Panel does not provide an easy way to configure a site.

Additional Requirements

• By default the / root page should show a nice Index/ Gallery page with images. Related to https://github.com/holoviz/panel/issues/2313.
• But it should also be able to serve a custom landing page at "/" and specify an alternative location like "/gallery" for the Gallery page.
• By default apps should include a nice menu to enable easy navigation across the site. Related to https://github.com/holoviz/panel/issues/2314

Additional Context

The ability to easily serve a site from a list or glob of scripts is currently something special to Panel and should be treasured. Streamlit and Dash cannot do it. Bokeh and Voila don't have the nice gallery page that Panel now has.

• Deployments can be hard. So if you can keep to one deployment and add more apps via a configuration file it can make your time to market much faster.
• In my experience a single, micro app is not enough. Things quickly grow into multi-page apps/ larger sites.
• This can save energy resources and help make the world greener. For example deploying apps individually often means allocating a minimum of resources per app. In a Trading Environment where I am we can quickly create 10s, 100s or 1000s of apps that might not be used that often. But still needs to be available. Thus we could be running a lot of resources idle.

I would like to specify something like panel serve site.toml and then the site is created from that with menu and nice gallery file. My current attempt to make it easy for Panel users to do this is here https://github.com/MarcSkovMadsen/awesome-analytics-apps-template

[philippjfr]

This makes a lot of sense to me. The configuration file should also support all the commandline options that you'd usually supply to panel serve. We should come up with a full specification of what the configuration file would look like and all the options it would support.

[jbednar]

BTW, there's a current proposal to add .ini support to Param: https://github.com/holoviz/param/issues/470 . I'm meeting with the author in a few days to plan how to incorporate those changes.

[philippjfr]

BTW, there's a current proposal to add .ini support to Param: holoviz/param#470 . I'm meeting with the author in a few days to plan how to incorporate those changes.

Might come in useful but not that all the CLI commands are inherited from Bokeh and therefore are not param based.

[jbednar]

Sure, but isn't this newly proposed functionality? Seems like we can accept a file and do what we want with it.

[MarcSkovMadsen]

I believe the following formats could be used for the configuration

• .ini
  • Pro: No need for additional dependencies as ConfigParser is built into python
  • Cons: Very simple format that does not support nesting.
• .toml:
  • Pro:
    • Would enable Panel to store it's configuration in the pyproject.toml file.
    • I have experience storing user and application data in .toml files and have existing code to build on.
  • Cons:
    • Adds additional toml package dependency
• .yaml:
  • Pros:
    • Used widely outside Python for configuration
    • Used by conda in the environment.yml and anaconda-project.yml. files
    • Already a dependency and used by Bokeh.

We need nested configuration. This rules the .ini format out. In principle we can support both .toml and .yaml format.

What would you prefer @philippjfr ?

I will continue using the yaml format in the PR until I get some feedback.

UPDATE: I initially wrote that yaml adds additional dependency. That is not true as bokeh depends on PyYAML. I've updated above.

[MarcSkovMadsen]

And example configuration file could look like

- environment:
    BOKEH_LOG_LEVEL: debug
- users:
    panel:
      name: HoloViz Panel
      url: https://panel.holoviz.org
      avatar: https://panel.holoviz.org/_static/logo_stacked.png
    marcskovmadsen:
      name: Marc Skov Madsen
      url: https://datamodelsanalytics.com
      email: marc.skov.madsen@gmail.com
      avatar: https://avatars.githubusercontent.com/u/42288570
      resources:
        github: https://github.com/MarcSkovMadsen
        discourse: https://discourse.holoviz.org/u/marc
    philippjfr:
      name: Philipp Rudiger
      url: https://github.com/philippjfr
      avatar: https://avatars.githubusercontent.com/u/1550771
      resources:
        github: https://github.com/philippjfr
        discourse: https://discourse.holoviz.org/u/philippjfr
- applications:
    stocks_altair:
      name: Stocks Altair
      servable: examples/gallery/apis/stocks_altair.ipynb
      url: stocks-altair
      author: philippjfr
      owner: panel
      description: An interactive stock price visualization app using Altair and YFinance
      tags:
        - Altair
        - YFinance
      resources:
        code: https://github.com/holoviz/panel/blob/master/examples/gallery/apis/stocks_altair.ipynb
    stocks_hvplot:
      name: Stocks HvPlot
      suspend: true
      servable: examples/gallery/apis/stocks_hvplot.ipynb
      url: stocks-hvplot
      author: philippjfr
      owner: panel
      description: An interactive stock price visualization app using HvPlot and YFinance
      tags:
        - HvPlot
        - YFinance
      resources:
        code: https://github.com/holoviz/panel/blob/master/examples/gallery/apis/stocks_hvplot.ipynb

If the servable path is not specified. Then nothing is served. But the meta data is still available for the index/ gallery page and the menu. Making it possible to include links to other apps (Streamlit, Dash, Voila, Flask etc) or resources (https://panel.holoviz.org) on your site. I.e. being able to build a larger analytics apps site with no dead ends.

The path specified in servable should be relative to the configuration file. Thus making it possible to store it in the same folder as the .py or .ipynb file, in the root of the project, elsewhere or in combinations here of.

If the suspend variable is set to true the item is skipped.

[MarcSkovMadsen]

You would be able to serve using

panel serve config.yml panel serve config1.yml config2.yaml panel serve pyproject.yml panel serve config.yml extra_apps/app.py extract_aps/app.ipynb

and similar combinations. Serving from a single config.yml file is important for example when serving and app using ContainDS. Here you can only specify the the path to one file. My guess is it is similar on other "DataScience" platforms. For example DataIku that supports serving a Bokeh app.

Also being able to specify environment variables is again important for serving in constrained environment like ContainDS where you cannot specify environment variables or flags. But only the path to a file.

[MarcSkovMadsen]

You would be able to find the meta/ config data at

panel.application.users panel.application.applications

or what ever you suggest @philippjfr ??

[jbednar]

I'd lean towards YAML because we are using it heavily in Lumen, and that way a Lumen author wouldn't have to learn another language and could configure both individual apps and the whole site the same way.

[MarcSkovMadsen]

I can see that Bokeh supports a bokeh.yaml file.

This leads me to some questions

• Can and should we use/ extend this mechanism?
  • Should we use the bokeh.yaml file or a panel.yaml file
• I would rather support something like panel serve forecasting/apps.yaml technical/analysis_apps.yml gallery.py because it keeps things simpler.

UPDATE: After having looked at the docs and code the bokeh.yaml file appears to be aimed at configuring settings only. I think that makes it inappropriate for extending to this use case.

[leycec]

For the love of all that is data science, please embrace YAML.

TOML is functionally, semantically, and syntactically inferior to YAML in every possible way. TOML is something that should have never happened and certainly never should have been adopted as the official standard for declaring Python project metadata. Everything outside the Python microcosm and even most things within the Python microcosm (excluding PEP 518) leverage YAML.

YAML is the way. Friends don't let friends TOML.

[itsgifnotjiff]

I would love to see a simple .yml to configure Awesome-Panel like templating for the Multiapp Index page. Is there an effort I can contribute to or is this not a priority as of Panel 0.14.X?

If it is not a priority, is there a resource to walk me through how @MarcSkovMadsen created the Awesome Panel App Gallery?