Add API

[robmarkcole]

Just an idea https://ndres.me/post/jupyter-notebook-rest-api/

[SimonBiggs]

What's a use case this would be trying to achieve?

[robmarkcole]

To expose the functions within the notebooks to other services. For example, we have internal web page that displays data feeds on lab temperature etc, and this way those web pages could use the python functions. Also EXCEL can call rest API's so this could solve a longstanding headache of sharing python work with my EXCEL only colleagues

[SimonBiggs]

I guess my aims for this particular project are the following:

• Do one thing and do it well
• Make it so that others, even excel types, can open up the form GUI markdown file and work out roughly what it's doing and make small changes if need be
• Make the time cost of creating a usable GUI so small that most pythons scripts can have a GUI further opening the tools up to colleges, especially excel types.

I still don't fully get that you're trying to say, but I don't see how it fits into this project.

[SimonBiggs]

Are you saying, making a tool that makes making APIs easy?

[SimonBiggs]

Or are you saying making it so that whenever there is a form open, it can be accessed via API?

[SimonBiggs]

... I think I'm starting to see what you mean ... It might be worthwhile ...

[robmarkcole]

I totally appreciate and respect the aims of this project. My proposal comes from the experience of having access to several excellent web-GUI applications at work that have no way to programatically interact with. For example we have a web app for entering data into one of our databases, works fine but people want to prep the tabular data in excel then paste into the GUI. However the GUI doesn't allow pasting of tables so they end up pasting cell-by-cell which leads to lots of frustration. I think it would be nice to bake in an API

[SimonBiggs]

I think that post shows that it is already quite easy to create an API with the current notebook. It's quite a nice feature I didn't know about.

I think if someone wants to make an API in that way they should use that method. Does that sound reasonable?

[SimonBiggs]

That makes sense. So what would baking in an API look like?

[robmarkcole]

Putting myself in the shoes of the average python user, I think they would probably like to just add functions to a .py file which is imported to their notebooks, and have those functions accessible via the api. That way they could say to a colleague 'I've just added a function for XYZ, heres the docs'

[SimonBiggs]

The way the form is set up any API usage would automatically live update the visuals, so that will be quite nice.

[SimonBiggs]

What might it look like in the markdown file?

[robmarkcole]

I would need to spend more time with this package before I can give an example. Agreed being able to monitor usage would be useful

[SimonBiggs]

The more I think about it, the more it is a great idea.

In the future I want to build more intricate things on top of this package and that would open up those possibilities.

Potentially there could be an API for each variable individually, an API for the current form being viewed being able to update the URL, an API to pull all of the results, and an ability to write something like:

<section-api name=something input=something output=something>

```python
python code that runs on api call
```

</section-api>

[SimonBiggs]

The fact that the person using the API can also open the form and see the results helps make the API be much easier to understand...

[robmarkcole]

I think key to the success of any product/project is turning it into a platform that others can extend

[SimonBiggs]

If initially, just to open it up enough for you to start fiddling with it, I made it so that the api looked a little like this:

http://localhost:8888/scriptedforms-api?name=variablename&value=variablevalue

[SimonBiggs]

Maybe just for the ability to keep it around in the future it might look like:

http://localhost:8888/scriptedforms-api/v1/variable?name=variablename&value=variablevalue

[SimonBiggs]

Actually, the form name needs to be in there, so:

http://localhost:8888/scriptedforms-api/v1/form-name.md/variable?name=variablename&value=variablevalue

[SimonBiggs]

And the Jupyter Notebook token too, so in fact the minimal version would have to look something like the following:

Start server like so:

scriptedforms form-name.md --token='a_custom_token'

Then the API would look like just a standard HTTP GET (in reality to be proper REST API it should be POST, but will just use GET for the simple v1) of:

http://localhost:8888/scriptedforms-api/v1/form-name.md/variable?name=variablename&value=variablevalue&token=a_custom_token

[SimonBiggs]

How does that sound? If I just implemented that for now would that be workable?

I can make it so that I keep this really simple API around as a permanent v1 if you find it useful. As I make a better API I'll put it under 'scriptedforms-api/v2/'

[SimonBiggs]

Just so that forms under nested directories make more sense I think the following might be a bit better:

http://localhost:8888/scriptedforms-api/v1/variable/form-name.md?name=variablename&value=variablevalue&token=a_custom_token

[SimonBiggs]

Or what if, for now it was as simple as:

POST http://localhost:8888/scriptedforms/form-name.md HTTP/1.1 Content-Type: application/json { "token" : "a_custom_token", "python" : "code to run, such as setting a variable or running a function imported within the form, new lines would look like \n" }

So therefore it's the same web address, you're just running POST to it instead of GET.

Benefit of that is it covers most use cases with a single method.

[SimonBiggs]

Okay! :)

There is now available a really basic API that should be flexible enough to get started.

Pull the most recent git commit (https://github.com/SimonBiggs/scriptedforms/commit/e6fea2419f99abf5f5349f2699818c7afd86cfd9) and install from the source.

Create the following markdown file called quick-start.md

# An example

<section-live>

<variable-string>your_name</variable-string>

```python
print('Hello {}!'.format(your_name))
```

</section-live>

Write the following into a terminal

scriptedforms quick-start.md --token boo

Then write the following into Python:

import requests
requests.post('http://localhost:8888/scriptedforms-api/v1/code/quick-start.md?token=boo', data='your_name = "Simon Biggs"')

The name on the form should then change to Simon Biggs to agree with the Python code just run.

By the way, it probably won't update if you have your cursor within the text field on the form.

If you're happy with how this works, don't want to change it, and think that will keep you going for a while I'll send it up to pypi in the form of version 0.5.8. I won't guarantee that it won't change until I have documented it in one of the readmes available. For now consider it an undocumented feature subject to change.

If no issues arise I'll document it for the 0.6.0 release and lock it in.

[SimonBiggs]

So, the "code" API is implemented. I think I should probably just implement two more. Let me know what your thoughts are if you think there will be anything missing or if the interface could be improved.

So, any section will be able to have an extra parameter added to it called "api". The value of the api parameter becomes what needs to be passed to the section api POST. That section will then run when that section api is called.

In the markdown file it might look like <section-button api="api_name"> then to make that section run you might do something like:

import requests
requests.post('http://localhost:8888/scriptedforms-api/v1/section/quick-start.md?token=boo', data='api_name')

The third api is a GET api that simply gets the variable values. Something like

import requests
variables = requests.get('http://localhost:8888/scriptedforms-api/v1/variables/quick-start.md?token=boo')

[SimonBiggs]

@robmarkcole I'm going to get a chance to work on these further tonight. Let me know if you have any feedback on the above.

[robmarkcole]

@SimonBiggs looks like good progress, will take for a spin. Just posted an issue

[robmarkcole]

Tried

(scriptedforms-env) Robins-MacBook-Air:scriptedforms robincole$ scriptedforms quick-start.md --token boo

and got:

which results in errors on accessing:

Trying your suggested token gives:

[SimonBiggs]

Only the first API has been implemented (running code):

https://github.com/SimonBiggs/scriptedforms/issues/77#issuecomment-368221875

I haven't yet set up sections or variables. Working on them now.

[robmarkcole]

OK the basic is working then :)

[SimonBiggs]

You beauty :) awesome :)

[SimonBiggs]

Just working out some finer websocket details and I should be able to have those final two APIs up and running.

[SimonBiggs]

@robmarkcole Okay! :)

All three APIs are now available. Pull the new repo master, rerun pip install -e . --upgrade, then start a new server like so:

scriptedforms example/example/detailed.md --token boo

Then run the following python code:

import requests
import json
from IPython.display import display

result = requests.get('http://localhost:8888/scriptedforms-api/v1/variables/detailed.md?token=boo')
display(json.loads(result.text))

result = requests.post('http://localhost:8888/scriptedforms-api/v1/section/detailed.md?token=boo', data='submit')
display(json.loads(result.text))

result = requests.post('http://localhost:8888/scriptedforms-api/v1/code/detailed.md?token=boo', data='plt.plot(data)')
display(json.loads(result.text))

Be aware that running a sections code that has output does not have that output display within the form. I'm not sure how I would be able to do that. It might be worth me thinking about.

Take it for a spin and let me know what feedback you have :).

Kk, sleep time for me now :).

PS: As something really cool! Try running the following in a jupyter notebook:

import requests
import json
from IPython.display import display, display_png

result = requests.post('http://localhost:8888/scriptedforms-api/v1/code/detailed.md?token=boo', data="plt.plot([1,2,3,1], 'o')")
png = json.loads(result.text)[1]['image/png'][:-1]
display_png(png, raw=True)

[robmarkcole]

All working, nice work 🥇

[SimonBiggs]

:) good to hear :). Now just need an API to change the browser url, and investigate whether or not I can make the outputs display with the API.

[SimonBiggs]

Also, for now, I've decided not to document this anywhere. I think until I have more organised docs detailing how to use this will be overly complicated and potentially detract from the main purpose.

Soon I'm going to have docs up at http://scriptedforms.com.au, once that's the case I can have a section in there called API.

[SimonBiggs]

@robmarkcole could you do me a huge favour. At the moment I have two testing folders, tests-e2e and tests-unit. Would you be able to make a third folder, tests-api and then using the nose tests python package could you create automated tests for each of the implemented APIs?

Make sure you cover the uses for which you'll be using the api for so that I never break your applications.

That would be amazing :)

[robmarkcole]

Re docs, I really like the readthedocs site for pylinac, is that the tool you are going to use, then just use your own domain name?

Re tests yes happy to. By 'nose' tests (unfamiliar with that term) do you mean using the Nose package?

[SimonBiggs]

I wasn't going to use sphinx. Given the nature of the project I was actually going to give live examples. Have the markdown on the left, and then the resulting form on the right. I was then going to mimic the python within the examples using JavaScript so that the docs would work as is.

That way the docs would be a sort of live example of ScriptedForms itself.

[SimonBiggs]

I did mean the nose testing suite. But actually I'm not too fixed on the nose package. If you have a python testing framework that you prefer feel free to use that.

[SimonBiggs]

And thankyou by the way. That would be an awesome help.

[robmarkcole]

OK your implementation of the docs sounds really cool :)

I use unittest usually but would also consider pytest - any preference?

[SimonBiggs]

Would it be okay to use pytest? Just because that was the path that jupyterlab appears to have gone down and I prefer to align with their choices where possible given the infrastructure is similar:

https://github.com/jupyterlab/jupyterlab/blob/master/jupyterlab/tests/test_jupyterlab.py

[robmarkcole]

OK sure, seems the trend is towards pytest generally

[SimonBiggs]

Then that sounds like the way to go.

[SimonBiggs]

Thanks :)

[SimonBiggs]

Api has been added. Tracking the testing of the rest API in

https://github.com/SimonBiggs/scriptedforms/issues/88