Unable to use FsLab package with IfSharp notebook

[jancura]

Description

I have tried to do FsLab tutorial example with Azure Notebook, as well as locally with my Jupyter installation and IFSharp as kernel. However, FsLab.fsx is not able to reference the package dependecies.

Repro steps

I run in the cell following

1. load "Paket.fsx"

   Paket.Package [ "FsLab" ]

   load "Paket.Generated.Refs.fsx"

   load "packages/FsLab/FsLab.fsx"

2. The rest of the example either does not produce any results or it reports an error that it cannot reference dependencies listed in FsLab.fsx

Expected behavior

See the tutorial example.

Actual behavior

No results or reference errors.

[tpetricek]

What error are you seeing?

[jancura]

I got this error using Azure Notebook just after #load "packages/FsLab/FsLab.fsx":

The search directory '/home/nbcommon/IfSharp/bin/packages/FsLab/../Deedle.1.2.5/lib/net40' could not be found
The search directory '/home/nbcommon/IfSharp/bin/packages/FsLab/../Deedle.RPlugin.1.2.5/lib/net40' could not be found
The search directory '/home/nbcommon/IfSharp/bin/packages/FsLab/../FSharp.Charting.0.90.14/lib/net40' could not be found
The search directory '/home/nbcommon/IfSharp/bin/packages/FsLab/../FSharp.Data.2.3.2/lib/net40' could not be found
The search directory '/home/nbcommon/IfSharp/bin/packages/FsLab/../MathNet.Numerics.3.13.1/lib/net40' could not be found
The search directory '/home/nbcommon/IfSharp/bin/packages/FsLab/../MathNet.Numerics.FSharp.3.13.1/lib/net40' could not be found
The search directory '/home/nbcommon/IfSharp/bin/packages/FsLab/../DynamicInterop.0.7.4/lib/net40' could not be found
The search directory '/home/nbcommon/IfSharp/bin/packages/FsLab/../R.NET.Community.1.6.5/lib/net40' could not be found
The search directory '/home/nbcommon/IfSharp/bin/packages/FsLab/../R.NET.Community.FSharp.1.6.5/lib/net40' could not be found
The search directory '/home/nbcommon/IfSharp/bin/packages/FsLab/../RProvider.1.1.20/lib/net40' could not be found
The search directory '/home/nbcommon/IfSharp/bin/packages/FsLab/../Suave.1.1.3/lib/net40' could not be found
The search directory '/home/nbcommon/IfSharp/bin/packages/FsLab/../XPlot.Plotly.1.4.2/lib/net45' could not be found
The search directory '/home/nbcommon/IfSharp/bin/packages/FsLab/../XPlot.GoogleCharts.1.4.2/lib/net45' could not be found
The search directory '/home/nbcommon/IfSharp/bin/packages/FsLab/../XPlot.GoogleCharts.Deedle.1.4.2/lib/net45' could not be found
The search directory '/home/nbcommon/IfSharp/bin/packages/FsLab/../Google.DataTable.Net.Wrapper.3.1.2.0/lib' could not be found
The search directory '/home/nbcommon/IfSharp/bin/packages/FsLab/../Newtonsoft.Json.9.0.1/lib/net40' could not be found
The search directory '/home/nbcommon/IfSharp/bin/packages/FsLab/../Deedle.1.2.5/lib/net40' could not be found
The search directory '/home/nbcommon/IfSharp/bin/packages/FsLab/../Deedle.RPlugin.1.2.5/lib/net40' could not be found
The search directory '/home/nbcommon/IfSharp/bin/packages/FsLab/../FSharp.Charting.0.90.14/lib/net40' could not be found
The search directory '/home/nbcommon/IfSharp/bin/packages/FsLab/../FSharp.Data.2.3.2/lib/net40' could not be found
The search directory '/home/nbcommon/IfSharp/bin/packages/FsLab/../MathNet.Numerics.3.13.1/lib/net40' could not be found
The search directory '/home/nbcommon/IfSharp/bin/packages/FsLab/../MathNet.Numerics.FSharp.3.13.1/lib/net40' could not be found
The search directory '/home/nbcommon/IfSharp/bin/packages/FsLab/../DynamicInterop.0.7.4/lib/net40' could not be found
The search directory '/home/nbcommon/IfSharp/bin/packages/FsLab/../R.NET.Community.1.6.5/lib/net40' could not be found
The search directory '/home/nbcommon/IfSharp/bin/packages/FsLab/../R.NET.Community.FSharp.1.6.5/lib/net40' could not be found
The search directory '/home/nbcommon/IfSharp/bin/packages/FsLab/../RProvider.1.1.20/lib/net40' could not be found
The search directory '/home/nbcommon/IfSharp/bin/packages/FsLab/../Suave.1.1.3/lib/net40' could not be found
The search directory '/home/nbcommon/IfSharp/bin/packages/FsLab/../XPlot.Plotly.1.4.2/lib/net45' could not be found
The search directory '/home/nbcommon/IfSharp/bin/packages/FsLab/../XPlot.GoogleCharts.1.4.2/lib/net45' could not be found
The search directory '/home/nbcommon/IfSharp/bin/packages/FsLab/../XPlot.GoogleCharts.Deedle.1.4.2/lib/net45' could not be found
The search directory '/home/nbcommon/IfSharp/bin/packages/FsLab/../Google.DataTable.Net.Wrapper.3.1.2.0/lib' could not be found
The search directory '/home/nbcommon/IfSharp/bin/packages/FsLab/../Newtonsoft.Json.9.0.1/lib/net40' could not be found

[cgravill]

It's looking for the packages with version numbers, Paket by default doesn't need to do this.

As a workaround, you don't actually need to call that helper to get packages referenced as #load Paket.Generated.Refs.fsx does it for you:

Fair warning, if you're following https://fslab.org/getting-started/ Foogle doesn't seem to be included in the FsLab package now so you'd need to do something like:

Paket.Package [ "Foogle.Charts" ]
#load "Paket.Generated.Refs.fsx"

after that, the world bank schema seems to be a bit different, @tpetricek have things changed since that was created? Perhaps creating a new notebook and adding a link from the FsLab docs would be useful?

[jancura]

Thanks for the workaround. It did work. However, now I have an issue with plotting. If I use the example from the website, I get the error:

Chart.LineChart
 ([ for y in 1985 .. 2012 ->
     string y,
       [ cz.``Gross enrolment ratio, tertiary, both sexes (%)``.[y]
         eu.``Gross enrolment ratio, tertiary, both sexes (%)``.[y] ] ],
  Labels = ["CZ"; "EU"])

The field, constructor or member 'LineChart' is not defined. Maybe you want one of the following:
   Line

If I change to Chart.Line then I get:

No overloads match for method 'Line'. The available overloads are shown below (or in the Error List window).
Possible overload: 'static member Chart.Line : data:seq<#value> * ?Labels:seq<string> * ?Options:Options -> GoogleChart'. Type constraint mismatch. The type 
    '(string * float list) list'    
is not compatible with type
    'seq<'a>'    
.
Possible overload: 'static member Chart.Line : data:seq<#key * #value> * ?Labels:seq<string> * ?Options:Options -> GoogleChart'. Type constraint mismatch. The type 
    '(string * float list) list'    
is not compatible with type
    'seq<'a * 'b>'    
.
Possible overload: 'static member Chart.Line : data:seq<#seq<'K * 'V>> * ?Labels:seq<string> * ?Options:Options -> GoogleChart when 'K :> key and 'V :> value'. Type constraint mismatch. The type 
    '(string * float list) list'    
is not compatible with type
    'seq<'a>'    
.
Possible overload: 'static member Chart.Line : data:seq<Series<'K,#value>> * ?Labels:seq<string> * ?Options:Options -> GoogleChart when 'K : equality and 'K :> key'. Type constraint mismatch. The type 
    '(string * float list) list'    
is not compatible with type
    'seq<Series<'a,'b>>'    
.
Possible overload: 'static member Chart.Line : data:Series<'K,#value> * ?Labels:seq<string> * ?Options:Options -> GoogleChart when 'K : equality and 'K :> key'. Type constraint mismatch. The type 
    '(string * float list) list'    
is not compatible with type
    'Series<'a,'b>'    
.

[tpetricek]

The sequence of "The search directory ... could not be found" messages is just warnings - sadly, I cannot find a reliable way of ignoring them. (We add more search paths than necessary to support both legacy NuGet client and Paket).

As for Foogle, this is ineed an obsolete library and we should fix FsLab docs - the new library is (much more complete) XPlot (https://tahahachana.github.io/XPlot/). Try something like (I might be getting this wrong though):

Chart.Line
 ( [ cz.``Gross enrolment ratio, tertiary, both sexes (%)``
     eu.``Gross enrolment ratio, tertiary, both sexes (%)`` ])
  .WithLabels(["CZ"; "EU"])

[jancura]

I used the following:

[ cz.``Gross enrolment ratio, tertiary, both sexes (%)``;
  eu.``Gross enrolment ratio, tertiary, both sexes (%)`` ]
|> Chart.Line
|> Chart.WithLabels ["CZ"; "EU"]

It does not give an error but the output cell is not a chart but: Out[ ]: XPlot.GoogleCharts.GoogleChart

So, it seems that IfSharp cannot embed figures produced by XPlot in Out cells.

[tpetricek]

Oh no, I think most of the modern demos are using Plotly (also via XPlot, but using different set of APIs)

[jancura]

I think with Plotly I have to have credentials and do login. Is there no simpler way of doing plotting in IfSharp without the need of an additional separate account for Plotly?

[cgravill]

We use the Plotly offline mode via XPlot so there's no requirement for an account.

You need to open a few more scripts to make it work, there's more details in the feature notebook (in the root of the repository) but the quick version is:

#load "XPlot.Plotly.Paket.fsx"
#load "XPlot.Plotly.fsx"
open XPlot.Plotly

then a stand-alone example chart:

["giraffes", 20; "orangutans", 14; "monkeys", 23]
    |> Chart.Bar
    |> Chart.WithLayout (Layout(title = "Basic Bar Chart"))
    |> Chart.WithHeight 300
    |> Chart.WithWidth 400

Note the reason for the scripts is we want to keep the notebook system separate from a given charting approach.

[jancura]

Thanks for the answer. I have some feedback as a newcomer. If you look at F# for Azure Notebooks sample library, the figures are not rendered. It seems the cells to load the few more scripts you mentioned above are not executed. After I made a local clone of the library and executed every cell, then everything rendered well.

Given this insights, I went back to run FSLab Tutorial example. The example given at getting-started seems outdated to the tutorial example that you get when you download the basic FsLab template. I followed more closely the tutorial from the template and I tried using XPlot.GoogleCharts.
However, XPlot.GoogleCharts.fsx is not correct as it does no correctly reference the library. I guess a typo in capital letter. So, I copied the script into the cell and corrected to reference the library. Then it worked. You can find the working example here, as I made my notebook public.

I think it would be nice to have a helper script that would install and setup FsLab environment with Xplot charting out-of-the-box in IfSharp notebook, e.g. FsLab.Setup.fsx or FsLab.fsx.

I can make fixes to the GoogleCharts.fsx and possibly add the setup script if you approve.

[cgravill]

Great glad its working for you. Yes we'd very much appreciate improvements. :-)

The lack of rendered charts is caused by a rendering restriction related to how the output is marked up. We're attempting to resolve this but it's ongoing see: #142

Ah the GoogleCharts casing issue was me. Someone asked about it a while back #118 but I generally work on Windows so missed the casing inconsistency on GoogleCharts. Thanks for the catch - I've committed a fix if you want to give it a try locally. (if it needs more fixes, they're very welcome too)

Something to help people use FsLab sounds great. It looks like we need a mixture of documentation updates on the FsLab side https://github.com/fslaborg/FsLab and perhaps helper scripts or sample notebooks on the Jupyter notebook hosts.

[jancura]

I can make the helper script, then a sample FsTurial Notebook and update getting-started documentation on FsLab. However, I need some guidance. How shall I commit the helper script into IfSharp repo? Do I just commit directly to master, or do you follow the shared repository model? I do not see any pull requests so I assume you do not follow the fork and pull model.

[cgravill]

We use the shared repository model, just efficient at getting pull requests merged. :-) Similar to yourself I didn't originate the project just started helping out on relevant fixes.

[jancura]

Hi, I think I miss something. I did clone the repository, created locally my branch, and then I wanted to push. But I got following error:

ERROR: Permission to fsprojects/IfSharp.git denied to jancura.
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.

So, I guess I do not have an access to push in this way.

[cgravill]

Yes, the repository is default restricted F# community members who have worked on the projects for longer. When starting to contribute here, I used Git fork and pull request approach: https://guides.github.com/activities/forking/ With that, you can create your own fork, commit there and make a pull request back into this repository.

Then someone, probably me, will review and discuss with the aim to getting the changes merged to master. I use the same approach for large changes still.

Oh and great to hear you've got some changes incoming! It really helps to have people with a fresh perspective making improvements too.

[jancura]

I tried to use the committed helper script on a fresh install of IfSharp and I found that the script fails. The reason is that after the IfSharp install you only get main.group.csx csharp scripts in ".paket\load" folder. Then the script fails because of Paket.Generated.Refs.fsx fails to load missing main.group.fsx.

I am curious if we are even able to load all the stuff in one script as we load a file which content is only filled by generating references after the preprocessor already loaded files' content (empty at the time of loading).

So it seems we will always have to have two scripts: Install using Paket (FsLab.Paket.fsx) and Setup references for all installed packages (FsLab.fsx), and execute them in different cells.

This also seems to follow the already established pattern, as I can see we always have a Paket script and then a setup script for the charting libraries.

[cgravill]

Yes, I've found you need to use a two script approach. When you execute all the instructions in one script we're not able to launch Paket first then do the referencing. It's a bit messier but the best approach found so far. It'd be great to have a better approach if you find anything.

[cgravill]

I'm going to close this issue for clarity. There's a mix of parts that have been fixed, some others that are part of other libraries, and there's a proposed enhancement to make a helper script for FsLab. Happy to take a pull request with the two script form for the helper script as and when someone has time to work on it.