documentation on how to package libraries for brython

[glyph]

If I want to use a package from PyPI in Brython, what do I do? Right now I am manually adding packages or symlinks to a fork of brython, but it would be nice to be able to use Pip for this.

Until Brython has Pip (or something like it), at least some documentation around conventions for how to write and use libraries would be nice.

[kikocorreoso]

Not all Python libs are Brython compatible. It wouldn't be beneficial to let people install whatever from Pypi if it is not working. Maybe a Brypi (Brython Pypi) and a dedicated Brip (Brython pip management package).

Brypi could be a list of user cheched packages and versions and Brip a tool that checks first that list, if the package is in the list, download it on site-pagckages of the Brython distribution.

Ideas? Thoughts?

[glyph]

Not all Python libs are PyPy compatible, or Jython compatible, or Python 3 compatible, or, for that matter, CPython compatible. Nevertheless, PyPI is the shared repository for everything we call "python". I would strongly advise that you use the same infrastructure, since operating a package index is pretty hard, and a lot of effort has gone into making PyPI reliable, fast, and secure.

[glyph]

You could add a new Trove identifier to specifically indicate Brython compatibility, and then have "brip" refuse to download or install things without that identifier unless a special option, say, --force, is passed.

[jsbueno]

As for package managing: I had not thought of using PyPi itself - but Glyph's enphatic suggestion makes sense. The registering of packages into PyPi does allow the script to register which interpreter versions it is compatible with - it is a matter of allowing "Brython" in that register, and specifying parameters to be used in Brython packages "setup.py" file. (Note that this setup.py" file would be used by distutils to upload the package into PyPi using a traditional command line Python (or eventually Brython over node.js :-) ) )

Also, if Brython does get a nice "setup.py" file, it can put in PyPi - that would allow one to install say, browser_invaders, by just typing "pip install browser_invaders" in a virtualenv. That would fetch both that code, and the latest stable Brython, and arrange them in a working directory structure ready to be served.

(Personally I like keeping Brython in a "brython/" folder relative to www root)

On 29 January 2015 at 07:50, Glyph notifications@github.com wrote:

You could add a new Trove identifier to specifically indicate Brython compatibility, and then have "brip" refuse to download or install things without that identifier unless a special option, say, --force, is passed.

— Reply to this email directly or view it on GitHub https://github.com/brython-dev/brython/issues/119#issuecomment-71996691.

[glyph]

@jsbueno - exactly. There are many different ways of arranging your brython files right now, and they are not all compatible with each other. If there were some pip idiom that worked with, let's say, data_files, that would make it a lot easier for new users to get started!

[PierreQuentel]

I agree that PyPI is the best choice. Brython is hopefully popular enough now to ask the PyPI team if they would consider adding a Trove classifier "Programming Language :: Python :: Implementation :: Brython"

Joao, would you be ok to take the lead on this task of handling Brython packages through PyPI ? I'm sure others on the Brython group would be willing to help

[jsbueno]

On 30 January 2015 at 14:53, Pierre Quentel notifications@github.com wrote:

I agree that PyPI is the best choice. Brython is hopefully popular enough now to ask the PyPI team if they would consider adding a Trove classifier "Programming Language :: Python :: Implementation :: Brython"

Joao, would you be ok to take the lead on this task of handling Brython packages through PyPI ? I'm sure others on the Brython group would be willing to help

Yes, I think I can do that - I am taking a read on distutils to find out the best way to describe a Brython package with a "setup.py" - and meanwhile I could contact the Pypi folks as well.

Since Brython even makes use of a "site-packages" folder, I think that is the way to go: have a modified setup.py that installs the pack in Brython's site-packages.

Is there any enviroment variable which might optionally point to Brython's root? (Like Python's "$VIRTUAL_ENV", not PYTHONPATH). If not, what about "$BRYTHON_ROOT" ? (it would point to the directory where bryhton.js is, and ordinary install would go under $BRYTHON_ROOT/Lib/site-packages")

js -><-

—

Reply to this email directly or view it on GitHub https://github.com/brython-dev/brython/issues/119#issuecomment-72232026.

[ghost]

One thing to keep in mind, is that Brython (is currently) just a subset of python3. The differences between Cpython 3 and Brython is smaller each day, but at this time, I am not sure it is a good idea to just assume that a package will work with Brython. Most packages eventually do work directly out of the box, but that is because we try to import a library/module, find what is not implemented (or broken) and then implement (fix it) in Brython.

I feel that if we provide a pip like tool, that is giving people permission to download whatever and run it in Brython. I believe one of these days, 99% of packages will just work (when downloading via pip), but today, I do not think Brython is mature enough to just throw any package at it. (IMHO)

I feel that once we get all/most of the standard library modules working, and that most of the unittests work, then we can provide a pip tool to the masses. (but if someone wants to create a pip like tool, and it is used by brython development staff for the time being, I'd be all for it.)

Am I too conservative?

Billy

[kikocorreoso]

Am I too conservative?

I don't think so, I think you are just cautious. Maybe we could maintain a document with links to tested libraries working in Brython and a pip like tool to install libraries contained in that document.

[ghost]

For now, I like Kiko's recommendation. we could create a tool named brip, and maybe have a "location" for these modules. As time goes by, and Brython supports 99% of CPython 3x, then we could modify it to point to pypi.

What do others think? Billy

[jsbueno]

I think such a tool could be alright - if it is needed at all - but that we might point at pypi right now - for Brython-specifc packages. Setuptools allows enough meta-information for even an ordinary setup.py file to "know" certain files are supposed to be served as static content and live with Brython.

Being able to install and use ordinary, not intenteded for Brython, Python modules is another history - - Any module containing no native code or ctypes calls, and making no I/O would potentially work (but for some bug fixes) - we just have to think of a way to signal the module should be installed in a Brython's file structure. (Maybe could be as simple as passing some install directory parameters to PIP).

Anyway, there are two different things going on: 1) a way to package and install Brython itself, and packages specifically designed to work with it (like Brython's Pygame currently in the source tree, the Browser Invader games, some html5-using widget and application toolkits, a package to easily make RPC calls to Python functions on the server side, and so on)

2) A way to install generic Python packages in Brython's structure.

I will be focusing on (1) as it is even needed in a way to be able to assert a file structure where packages available for Brython will end up. (Although, since there is a "site-packages" directory, it will likely be just making use of it for both types of packages).

js -><-

On 2 February 2015 at 23:55, Billy Earney notifications@github.com wrote:

For now, I like Kiko's recommendation. we could create a tool named brip, and maybe have a "location" for these modules. As time goes by, and Brython supports 99% of CPython 3x, then we could modify it to point to pypi.

What do others think? Billy

— Reply to this email directly or view it on GitHub https://github.com/brython-dev/brython/issues/119#issuecomment-72577805.

[glyph]

A brip tool might make sense as a thin wrapper around pip, so that all the options were set correctly to put stuff into a particular brython environment. Also to pre-compile any installed packages to .js, just like pip would normally pre-compile any installed packages to .pyc.

[ghost]

+1 placing the modules/packages in site-packages

I like @glyph 's idea, but here is an example that might cause some issues. What about brython custom packages of popular python packages such as pygame? pygame will be implemented very different for brython then for CPython. How would this work with pypi and pip?

[jsbueno]

On 3 February 2015 at 11:25, Billy Earney notifications@github.com wrote:

+1 placing the modules/packages in site-packages

I like @glyph https://github.com/glyph 's idea, but here is an example that might cause some issues. What about brython custom packages of popular python packages such as pygame? pygame will be implemented very different for brython then for CPython. How would this work with pypi and pip?

We name it "brygame" instead? :-)

But seriously - I think it would be nice for this kind of packages to have a prefix for the package name in PyPi (this does not affect the actual module names) - something like brython-pygame, brython-twisted, brython-tkinter .

It does not have to be "brython-", neither should it be mandatory - but something along that could be recommended.

— Reply to this email directly or view it on GitHub https://github.com/brython-dev/brython/issues/119#issuecomment-72649319.

[glyph]

packages to havea prefix for the package name in PyPi (this does not affect the actual module names) - something like brython-pygame, brython-twisted, brython-tkinter

"package name in PyPI" technically is "distribution name"

Also, it is possible that as Brython becomes more popular, PyGame (or other related packages) may want to actually support Brython in the main distribution! Although lots of the core of PyGame would be very different on Brython, it could perhaps be split into a private low-level backend interface and then a public high-level interface, with all utility and convenience logic written in pure, implementation-agnostic Python :).

I know that I would want to make brip install twisted do something useful :-).

[dstufft]

For what it's worth, if there's something pip can do to support brython better, as long as it doesn't end up regressing non Python things, there's a good chance we can get it in. I don't know what brip would actually need over pip (this was linked to me and I've never used brython) so maybe it's just too different to sanely fit into pip itself I don't know.

[dstufft]

Oh, and by the Way, wheel supports arbitrary implementations in the tag name, so you can have a compiled Wheel which only contains code for Brython. You'd get something like foo-1.0-brythonXY-none-any.whl or something like that. We'd probably have to teach tooling to understand it, but the framework is there to support "binary" artifacts which are specific to a particular implementation.

[olemis]

IMHO we do not need brip or anything alike . After the work made in #189 the steps to install libs should be similar to this :

• Create a Python virtual env in the server (optional)
• Install your libs (easy_install , pip , ...)
• Publish your virtual env folder via HTTP
• Add virtual env base URL to sys.path

Therefore I'm closing this ticket . Feel free to reopen it otherwise .

[olemis]

There is another approach consisting in publishing via HTTP .vfs.js files (like this one ) containing code for several modules . It is necessary to add VFS file URL in sys.path .

[glyph]

One of the problems with the approach listed above is that pip can't do its compilation step to ensure that everything is compiled to JS, in the way that it would normally compile everything to .pyc. (If brython shipped a node.js compatible compileall module, this would work with no changes to pip.)

Another problem is that it's still not documented, even if it should work this way :). So I think this ticket should be reopened until there's clear documentation that you are at least supposed to do things this way (pip install with the python runtime of your choice).

[vikrantsanghvi]

@glyph actually i am using brython for my data science project which is written in ipython jupyter notebook and the logic is in python. For getting it in web app form ,i am facing an error on the import of libraries like pandas which are basic ones for data sciences. it searches in src/Lib/site-packages/ but is not able to find. I believe because the package is not present in brython python standard library. How can i fork it to use it in my brython project? if you can provide some help or any documentation link or workaround, it would be great.

Thank you in advance. I am really looking forward to hearing from you regarding this query.

[kikocorreoso]

@vikrantsanghvi I think you are confusing the frontend and the backend. In the Jupyter notebook the python code runs in a backend kernel (the IPython kernel). The notebook is kernel agnostic, it means it can run Python, Julia, Haskell,..., code but using the same user interface (Jupyter) for all of them. In this context brython will run on the frontend (Jupyter notebook) and pandas et al data analytics related is running in the IPython kernel (backend).

I've made a library, brythonmagic, that runs in the Jupyter notebook and allows to write brython code in a code cell. It also allows to pass info from the backend to the frontend (no dataframes 'serialization'). The way to pass info is creating a string representation of the data that is injected in the brython code and the final brython code runs with the data from the backend.

In the repo linked above there are several notebooks showing how to work with brython in the notebook using brythonmagic.

I hope it helps.

[artem-v-shamsutdinov]

Hi,

We are in early stages of writing a cordova based federated server system, where every phone is it's own server with data specific for it's user. We are looking into running statistical analysis right on the device (on the data for it's user). I'm just starting to research our options but it looks like the only choice we have is bundling kivy along with cordova. Would compatibility with libraries like pandas be on the road-map in the foreseeable future?

Thanks, :)

[kikocorreoso]

Hi,

I think you are confused. If you use kivy you wouldn't need cordova. OTOH, brython is a python implementation written in javascript that allows you to run python on the frontend.

If you want to make a kivy app it is better to ask in their forums.

[artem-v-shamsutdinov]

Thank you for your quick response. Yep, it's pretty confusing at first. Basically we are going to be running a full server right in the browser and are treating the browser as an OS (a with database and everything else needed). And we want to run a stats module there as well, which means we need support for python, as far as I can understand it.

Thanks, :)

On Sun, Apr 30, 2017 at 12:55 PM, kikocorreoso notifications@github.com wrote:

Hi,

I think you are confused. If you use kivy you wouldn't need cordova. OTOH, brython is a python implementation written in javascript that allows you to run python on the frontend.

If you want to make a kivy app it is better to ask in their forums.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/brython-dev/brython/issues/119#issuecomment-298249923, or mute the thread https://github.com/notifications/unsubscribe-auth/AHwSnURv-zY1rCTxIdJhjB6eRxEqwCkNks5r1NkygaJpZM4DYxgu .

[kikocorreoso]

Pandas has compiled dependencies and it is not an easy target.

If your stats are simple maybe it would be better to use Java/android, Swift/iOS, Javascript/Cordova, Kivy/Buildozer,..., and create your own statistical functions or use something like sqlite sql operations to do the work.

Brython doesn't support pandas and neither is a backend technology.

[Ordiel]

any update on this?

I am trying to get lxml to work in Brython and in deed a package manager (pip) would be nice

[kikocorreoso]

lxml has compiled dependencies.

Brython can work with pure python packages and javascript libs. Try to use a pure python/js xml parser.

[glyph]

@kikocorreoso Technically you are of course correct, but it would be nice for brython to have some API-compatible way to manipulate XML that was based on browser APIs in the frontend but based on whatever back-end libraries were available on a native platform. (Maybe, one day, Rust code instead of likely-to-be-vulnerable C, but still.)

The nice thing here would be documentation for a project like lxml's maintainers (rather than their users, who obviously can't get it to work unless it's re-built to work) for how to run their tests on e.g. Travis-CI so that they could ship a Brython backend that could be installed as normal via pip. Although libraries can be distributed using normal python tools for Brython now, this is a far cry from many libraries actually working.

[stuaxo]

Would using Vendorize be a good step in the right direction ?

It can take python packages and leave the installer version in a directory of the users choice.

https://pypi.org/project/vendorize/

[glyph]

I think vendorize is for a much more specialized use case, since it asks you to replace all your imports with a sub-package, and that's not what we would want here.

[stuaxo]

Cheers, it was long ago enough that I had forgotten the details :)

[devuc]

@glyph actually i am using brython for my data science project which is written in ipython jupyter notebook and the logic is in python. For getting it in web app form ,i am facing an error on the import of libraries like pandas which are basic ones for data sciences. it searches in src/Lib/site-packages/ but is not able to find. I believe because the package is not present in brython python standard library. How can i fork it to use it in my brython project? if you can provide some help or any documentation link or workaround, it would be great.

Thank you in advance. I am really looking forward to hearing from you regarding this query.

[devuc]

same problem

[kikocorreoso]

@devuc brython is a Python3 transpiler. It transpiles python code to JS. If your Python code relies on C/C++/Fortran/Rust/... compiled code (like pandas) then you are not lucky and you are misunderstanding what Brython tries to solve (A Python 3 implementation for client-side web programming 😃 , Brython is designed to replace Javascript as the scripting language for the Web., 🍰 ). So, for instance, Brython can't help you to transpile from C to JS. 😞

If your code is 100% python based then Brython could be useful and interesting.

If you want to run pandas on the client side maybe it is better to have a look to projects like pyodide.

I hope it helps.

[stuaxo]

This may indicate a documentation gap. It could be good to have a note about this linked pretty close to the front page, about what works and what won't work.

I'm not one of those that would expect those things to work, but I think for beginners it's a pretty reasonable to wonder why some "python" libraries work and others don't - they may not know that many things they install via pip are not 100% python.

[kikocorreoso]

That is a good point.

In general I see the official front page is clear but I suppose I'm biased.

@PierreQuentel what do you think about adding some lines to the front page and/or some paragraphs in the docs in order to avoid this misconception?

[PierreQuentel]

@dstufft I agree, this is not mentioned clearly enough in the documentation. Would it be ok to add a Q/A in the FAQ page ?

[glyph]

FWIW, I wrote this ticket because I wanted a simple template for writing a tox.ini configuration or something to run unit tests on brython for small libraries to make sure that pip install-ing them in this way will work, and I'm still totally at a loss for how to do that :)

[rayluo]

A brip tool might make sense as a thin wrapper around pip, so that all the options were set correctly to put stuff into a particular brython environment. Also to pre-compile any installed packages to .js, just like pip would normally pre-compile any installed packages to .pyc.

FWIW, I noticed this conversation 6 months ago when I was also seeking solutions for packaging/installing Python library, so it was only appropriate to name my effort after the name brip which is based on Brython's VFS mechanism.

Right now, brip is still in its infancy. The usage pattern works the way it should, but many existing PyPI packages - even some pure Python ones - still won't work as expected. I suspect there might be some nuance for from . import foo in VFS. If @PierreQuentel could help me take a look from the VFS side, we may dramatically increase the compatibility of many more pure Python packages from PyPI.

[glyph]

we may dramatically increase the compatibility of many more pure Python packages from PyPI

This is great, and I don't mean to discourage it — but you're still depending on these packages to work by accident :). Lots of packages randomly import stuff or do things with os, sys, errno, fcntl, or other modules that aren't or will never be part of Brython, and need to add logic to avoid these things, and that means CI that tests with brython :)

[rayluo]

This is great, and I don't mean to discourage it — but you're still depending on these packages to work by accident :). Lots of packages randomly import stuff or do things with os, sys, errno, fcntl, or other modules that aren't or will never be part of Brython, and need to add logic to avoid these things, and that means CI that tests with brython :)

Thanks for your input. I won't be discouraged by that, because it is simply a fact which has been acknowledged in the "Limitations" section in brip. From a tooling perspective, brip could maintain a "compatible packages" list to provide better use experience, which was also an idea mentioned earlier in this thread. The goal of brip is not necessarily to bring the entire PyPI ecosystem to Brython, but it could facilitate the discovery and use of those packages that can work with Brython.

PS: As part of the brip effort outcome, it becomes more clear that what would work, what would not, and what could be done to convert those middle ground into Brython territory. Those would be a topic in the brip repo, not here.