Parity with existing Python-based component generation pipeline

[rpkyle]

Currently, components in Dash.jl are generated via macros, which populates the component template from within Julia. This differs from the way components are currently generated for Plotly's existing Dash backends -- these are created using (Python) scripts for Python and R alike.

plotly/dash#1197 proposes to add similar functionality for Julia component generation. Doing so would allow us to

• update Julia components at each release cycle
• standardize documentation and component generation practices
• permit non-Julia users to easily build Dash components for this user community
• implement the same structure we currently use for component build tests in other repositories

@waralex has noted that switching to pre-generated components may lead to scoping issues that require some refactoring within Dash.jl; specifically, we want to ensure that generated components have global scope, so that they are available to Dash apps once the module is loaded.

I've noticed that the Project.toml file is generated within Julia, but I'm curious if there's any way to obtain this metadata outside of Julia's REPL for component modules. This will be a minor obstacle as we work to create component libraries for Julia, since we try not to force Python users to do any component generation work in R, for example.

@mbauman if you have any suggestions related to scoping or generating Project.toml outside of Julia, your insights would be much appreciated 🙂

[waralex]

@mbauman Briefly, the problem is as follows. The main package needs to know which component packages were used and be able to read the list of necessary resources from them. In Python, this seems to be solved by filling in an array in the global scope, which is not an option for Julia. I haven't worked with reflection in Julia yet, so if you have a clue about the solution, I would be very grateful. If not, I will study how the Revise.jl works - it clearly knows how to get a list of used packages.

[waralex]

It looks like I've found a solution. The only thing it requires is for Dash to be included before component packages. The solution is to add a callback to Base.package_callbacks during initialization of Dash and check all the packages that are included after that for the presence of some characteristic method like get_dash_resouces. Spied from Revise.jl.
What do you think about this?

[mbauman]

Ah, interesting. So essentially https://github.com/plotly/dash/pull/1197 would create DashCoreComponents.jl and DashHTMLComponents.jl packages... and allow others to similarly create DashBootstrapComponents.jl and more.

And what you want is for the user to just be able to do:

using Dash

and automatically get access to the dcc_* and html_* (and potentially even dbc_*) functions... but only if they've manually Pkg.added DashBootstrapComponents?

Dynamically spying on what packages are available at init time is problematic with precompilation (and compilation) — which isn't a problem for Revise as it's an interactive package at heart.

[chriddyp]

Pkg.add'ed

I think the distinction is that the users should still do using DashCoreComponents. So, "installed" packages wouldn't be registered by Dash, only "imported" packages (which I'm assuming is "using")

Dash needs to know which packages are imported so that it can serve the necessary JavaScript & CSS files on page-load. These JS & CSS files are bundled with the component packages rather than with the core Dash library.

Dash only needs to know these things right before page-load, so this doesn't need to happen in the using DashCoreComponents step, it could happen later during runtime as long as it's before page-load.

[waralex]

Not exactly. Dash generates an index page that, among other things, should contain links to the resources used. First of all, js files. Accordingly Dash must know which component packages the user is going to use. Also for these files I need to register routing in Dash. In Python it looks like this:

import dash
import dash_core_components as dcc
import dash_html_components as html

When importing dashcore components and dash_html_components, an array is filled in in the global scope with elements like

        'relative_package_path': 'async-plotlyjs.js',
        'external_url': (
            'https://unpkg.com/dash-core-components@{}'
            '/dash_core_components/async-plotlyjs.js'
        ).format(__version__),
        'namespace': 'dash_core_components',
        'async': 'lazy'
    },

And data from this array is used when building the page and routing in dash.

The question is how to compile this list of resources from separate packages included in a specific script in Julia. In the most Python-like way possible.

[waralex]

Another option that I just came up with is to keep this array in the global Dash scope and populate it from component packages on __init__ calls

[mbauman]

I mean, the Dash object itself knows what it's holding onto. Couldn't each Component hold onto this metadata? It could just be a reference to a shared constant NamedTuple.

[waralex]

The problem is that it is difficult to understand the entire set of components used in advance, except by the list of packages used. Because when creating a layout, you can use some components and add others in response to a callback, and they can differ depending on callback arguments

If it weren't for the requirements of maximum similarity to Python, I would pass the dependencies to Dash directly

[waralex]

There is another way that I was thinking about while developing Dashboard as a separate package. Create a separate registry for component sets and the corresponding infrastructure in Dashboards -install_components(name), use_components(name). This would eliminate the need to make every change to the components through the Registrator (given that the component developers are not familiar with Julia). And in general, I don't really like the idea of keeping packages that are only needed for other packages to work. But I don't know how the Plotly team will react to this idea

[rpkyle]

I've just started learning Julia, but I'm curious if conditional loading of JavaScripts and other assets based on whether a package is attached (as in R) is an alternative.

This code doesn't run as-is because of a MethodError related to applying names to an array of ::Symbol (no time to explore further right now), but consider as pseudocode

imported = setdiff(names(Main, imported=true), names(Main))

asset_fns = String[]

for package in imported
    for fn_name in string(names(package))
        if ":some_dash_metadata_fn" == fn_name
            push!(asset_fns, fn_name)
        end
    end
end

I think imported will yield the list of attached packages for the session -- using this, I wonder if one could walk this package tree, and look for :some_dash_metadata_fn within each one. If so, seems possible to operate on asset_fns and load only those assets described by the matched functions. Just a thought, apologies in advance if this looks very strange in Julia. 😸

Julia probably provides for a simpler way of teasing out the assets provided by a given component library, in R we did it this way initially for compatibility with htmltools (and to avoid saving any proprietary, serialized data from within R):

https://github.com/plotly/dash-core-components/blob/master/R/internal.R

[waralex]

What if Dash is not used in Main directly? If it is used in another module? This solution should work, but it should go recursively through all modules in the Main and further into the depth of all modules imported in the Main

module MyDash
using Dash
using DashCoreComponents
end

In this case only MyDash will be in Main

[mbauman]

I see the point about dynamically adding a component from a callback.

The "proper" way to do module introspection like this is with Base.loaded_modules. The question, though, is which modules do you look for? How do you know a given module is a Dash component library? Do they need to register it with Dash.jl? It still feels a bit smelly to me. Rather than having Dash look for libraries, I think Alex's suggestion of doing it the other way around is a good idea and should be safe.

That said, I agree that it feels silly to register these component libraries as first-class Julia packages... but they're not wholly Artifacts as they do define Julia code. The idea of a special registry does have some sense to it, but I think it'll be more complication and effort — on all sides — than it's worth.

How much does the existing dash-generate-components script/workflow guide package authors through the steps of registering on PyPI/CRAN?

[rpkyle]

The "proper" way to do module introspection like this is with Base.loaded_modules. The question, though, is which modules do you look for? How do you know a given module is a Dash component library? Do they need to register it with Dash.jl? It still feels a bit smelly to me. Rather than having Dash look for libraries, I think Alex's suggestion of doing it the other way around is a good idea and should be safe.

I suggested to @waralex that the R approach was not ideal (for reasons which were expressed earlier), but in the end it doesn't matter how it's handled in Julia, only that we provide a mechanism for loading only those JavaScript bundles that are necessary for the current app. So while dashCoreComponents, dashHtmlComponents and dashTable are all core Dash libraries, we only load the JS required for the ones you've imported prior to creating your app object. So if there's a better way to do this in Julia, that's great.

That said, I agree that it feels silly to register these component libraries as first-class Julia packages... but they're not wholly Artifacts as they do define Julia code.

As for whether or not to contribute them to Julia's package registry, if contributing Dash.jl as a package does not require that (core) component libraries also be registered as first-class packages, it may be a non-issue.

In the R world, we need to ensure that all the core libraries are available via CRAN, because Dash itself requires them -- even if no end user is likely to use these packages in isolation. It's not silly in that sense per se, but we're learning that the ecosystem works a bit differently on the Julia side, I think.

How much does the existing dash-generate-components script/workflow guide package authors through the steps of registering on PyPI/CRAN?

Maybe I'm not following your question, but dash-generate-components doesn't guide the package submission process at all. I continue to work on adding features to the generator to help produce CRAN-friendly packages, but in the end a component library developer is responsible for manually proceeding with submission. I believe the same is true for Python, but @Marc-Andre-Rivet can confirm.

[waralex]

@mbauman, I would like to consult with you on a couple more issues related to generation.

• Theoretically, the component parameters have a type restriction that is checked on the js side of the code, and this information is available at the time the component code is generated. Should I check these restrictions on the Julia side? I once looked that there are many restrictions that look like " Union{String, Vector{String}, Dict{String,String}, Vector{Dict{String, String}}}" and decided that such restrictions on the Julia side will be more of a hindrance than a help. Maybe I'm wrong, what is Your opinion?

• What do you think is better to use to automatic convert the Julia type to a Component when passing it to children parameter of another component? Base.convert(Component, a:: T) or define something inside Dash (now it's Dash.Front.to_dash (:: T), but it will at least change its name). The most obvious application of this is to pass the PlotlyBase structure as a component. it is possible that other similar cases will occur. I tend to think that Base.convert is the most understandable and logical solution... On the other hand Dash.Front.to_dash (:: T) is more flexible and can be used when passing any parameter, not just children. The physical meaning of it will be "converting an object before transferring it to a dash"

[mbauman]

Regarding asserting ::Union{String, Vector{String}, Dict{String,String}, Vector{Dict{String, String}}} — I think it'd end up being a loss of functionality. The important thing is that it can serialize via JSON2 to one of those things... and I think we can just lean on duck typing to trust that it does. That's pretty much everything JSON supports (except numerics)... do we get sensible errors if we surprise Dash by responding with something outside of that spec?

Right now to_dash(x) doesn't always return a Component — which it must if you decide to use convert(Component, x). I think it's fine (and good) to have a hook like to_dash that enables an easy place for folks to override behaviors. Really the only reason to use convert is that it'd automatically get called upon setfield!/setindex!/default-constructor use.

@rpkyle Yes, you answered my question exactly as I intended. On the Julia side, things are very automatable and we could, e.g., automatically create/push-to a GitHub repo and request registration.

[rpkyle]

@rpkyle Yes, you answered my question exactly as I intended. On the Julia side, things are very automatable and we could, e.g., automatically create/push-to a GitHub repo and request registration.

@mbauman Amazing. Julia is a large paradigm shift for me, but so far there's a lot to like.

(Also, offline Marc-André confirmed that nothing automated is happening on the PyPI end, which is what I suspected. On the other hand, we welcome community PRs, in case this is a feature Julia users would appreciate. 🙂)

[waralex]

do we get sensible errors if we surprise Dash by responding with something outside of that spec

Yes, they should be from the frontend, and in the server logs (when I implement them) at least in debug mode. I know that this is my problem in working with Julia - when I leave duck typing my c++ experience screams "don't do it, at least do it with template magic". I'm fighting it :)

automatically create/push-to a GitHub repo and request registration.

I would love to see an example of such an implementation. My experience in Julia is quite small and I'm interested in looking at things in domains that I haven't encountered yet.

[rpkyle]

@alexcjohnson Is it OK to close this issue now that plotly/dash#1197 has been merged? Are we awaiting anything else at this point?