Ability to install addons

[joebo]

The playground has some packages enabled, shown below, which were the packages required to run plot.

Running jpkg results in an error

   (0!:100) <'jlibrary/system/util/pacman.ijs'

   'install' jpkg 'convert/misc'
Package Manager will run in read-only mode, as you do not have access to the installation folder.

To run as root, open a terminal and use sudo to run J.
The addons directory does not exist and cannot be created.

It is set to: /addons/.

You can either create the directory manually, or set a new addons directory in your profile script.

file permission error

Playground Link

System analysis and design work is needed to determine how to enable package installation.

Some monkey patching gets farther along in the installation process. My thinking is we can override httpget to make a javascript call to download

   (0!:100) <'jlibrary/system/util/pacman.ijs'

   HASFILEACCESS=: 1
   HASADDONSDIR=: 1

   SystemFolders_j_  =: SystemFolders_j_ , ('install';'')

   testaccess_jpacman_ =: 3 : 0
f=. <jpath'~install/testaccess.txt'
  '' 1!:2 f
  NB. 1!:55 f
  1
)
   testaccess_jpacman_''
1

   (1!:0) 'addons/*'
+--------+-----------------+----+---+------+----------+
|stats   |2022 5 23 22 0 36|4096|rwx|----d-|drwxrwxrwx|
+--------+-----------------+----+---+------+----------+
|arc     |2022 5 23 22 0 36|4096|rwx|----d-|drwxrwxrwx|
+--------+-----------------+----+---+------+----------+
|config  |2022 5 23 22 0 36|4096|rwx|----d-|drwxrwxrwx|
+--------+-----------------+----+---+------+----------+
|general |2022 5 23 22 0 36|4096|rwx|----d-|drwxrwxrwx|
+--------+-----------------+----+---+------+----------+
|math    |2022 5 23 22 0 36|4096|rwx|----d-|drwxrwxrwx|
+--------+-----------------+----+---+------+----------+
|graphics|2022 5 23 22 0 36|4096|rwx|----d-|drwxrwxrwx|
+--------+-----------------+----+---+------+----------+

   'install' jpkg 'convert/misc'
The addons directory does not exist and cannot be created.

It is set to: /addons/.

You can either create the directory manually, or set a new addons directory in your profile script.

|value error: e
|       (_1-:msg)+.0=#msg=.e
|[-0]

   install 'github:jpjacobs/types_dict'
|value error: pmview_applycounts
|   'num siz'=.    pmview_applycounts dat
|[-0]

   httpget =: {{0;'foo'}}
httpget 'www'

SystemFolders_j_
httpget =: {{0;'foo'}}
   httpget 'www'
+-+---+
|0|foo|
+-+---+

   SystemFolders_j_
+-------+--------+
|temp   |        |
+-------+--------+
|addons |/addons/|
+-------+--------+
|install|        |
+-------+--------+

Playground link

Goals:

1. Be able to install standard addons
2. Be able to install addons from github (e.g. install 'github:AshyIsMe/jreference' or install 'github:jpjacobs/types_dict')

This will allow users to try out different addons without impacting their local J system

Currently loaded packages:

   ,. Loaded_j_
+-------------------------------------+
|//jlibrary/system/util/scripts.ijs   |
+-------------------------------------+
|/addons//graphics/afm/afm.ijs        |
+-------------------------------------+
|/addons//graphics/color/colortab.ijs |
+-------------------------------------+
|/addons//graphics/bmp/bmp.ijs        |
+-------------------------------------+
|/addons//arc/zlib/zlib.ijs           |
+-------------------------------------+
|/addons//graphics/png/png.ijs        |
+-------------------------------------+
|/addons//math/misc/trig.ijs          |
+-------------------------------------+
|/addons//graphics/plot/jzplot.ijs    |
+-------------------------------------+
|/addons//graphics/plot/plot.ijs      |
+-------------------------------------+
|/addons//graphics/viewmat/viewmat.ijs|
+-------------------------------------+

[bobtherriault]

Very cool use of the Playground if it were to allow preview of packages on a browser, prior to using on your main system.

Excellent work, Joe

Cheers, bob

On May 23, 2022, at 18:22, joebo @.***> wrote:

The playground has some packages enabled, shown below, which were the packages required to run plot.

Running jpkg results in an error

(0!:100) <'jlibrary/system/util/pacman.ijs'

'install' jpkg 'convert/misc' Package Manager will run in read-only mode, as you do not have access to the installation folder.

To run as root, open a terminal and use sudo to run J. The addons directory does not exist and cannot be created.

It is set to: /addons/.

You can either create the directory manually, or set a new addons directory in your profile script.

file permission error

Playground Link

System analysis and design work is needed to determine how to enable package installation.

Goals:

• Be able to install standard addons • Be able to install addons from github (e.g. install 'github:AshyIsMe/jreference' or install 'github:jpjacobs/types_dict') This will allow users to try out different addons without impacting their local J system

Currently loaded packages:

,. Loadedj +-------------------------------------+ |//jlibrary/system/util/scripts.ijs | +-------------------------------------+ |/addons//graphics/afm/afm.ijs | +-------------------------------------+ |/addons//graphics/color/colortab.ijs | +-------------------------------------+ |/addons//graphics/bmp/bmp.ijs | +-------------------------------------+ |/addons//arc/zlib/zlib.ijs | +-------------------------------------+ |/addons//graphics/png/png.ijs | +-------------------------------------+ |/addons//math/misc/trig.ijs | +-------------------------------------+ |/addons//graphics/plot/jzplot.ijs | +-------------------------------------+ |/addons//graphics/plot/plot.ijs | +-------------------------------------+ |/addons//graphics/viewmat/viewmat.ijs| +-------------------------------------+

— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you are subscribed to this thread.

[joebo]

Very cool use of the Playground if it were to allow preview of packages on a browser, prior to using on your main system.

Thanks bob. I also think it would be a great way for package authors to create some demos to raise interest in their packages too in a non-intrusive way. I can imagine on github a package author would link to the playground in their readme to try it out online in the playground.

[bobtherriault]

I was expecting to be surprised at what running J on a browser was going to enable, but I am surprised at how many times that I have been surprised. This is turning into a very interesting application.

Congratulations to all who have been involved in this project. Your work is appreciated.

Cheers, bob

On May 23, 2022, at 19:03, joebo @.***> wrote:

Very cool use of the Playground if it were to allow preview of packages on a browser, prior to using on your main system. Thanks bob. I also think it would be a great way for package authors to create some demos to raise interest in their packages too in a non-intrusive way. I can imagine on github a package author would link to the playground in their readme to try it out online in the playground.

— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you commented.

[joebo]

Adding some research notes here... There are several challenges to overcome:

For github repos, github does not support cross origin requests on the zipball url... As shown here:

 (0!:100) <'jlibrary/system/util/pacman.ijs' 

httpgetJS =: {{)n
    let url = jgetstr("URL_httpget_");
    var request = new XMLHttpRequest();
    request.open('GET', url, false);  
    request.send(null);
    console.log(url);
    if (request.status === 200) {
      console.log(request.responseText);
      jsetstr("RESPONSE_httpget_",request.responseText);
    }

}}

NB. patch getpackageurl to use a different API which supports CORS
getpackageurl_jpacman_ =: 3 : 0
'tag rep cmt'=. y
select. tag
case. 'github' do.
  ball=. IFUNIX pick 'zipball';'tarball'
  p=. <;.2 rep,'/'
  rpo=. ;2 {. p
  'https://github.com/',rpo,ball,'/',cmt
case. do.
  ''
end.
)

NB. maybe we need to overwrite httpget?
httpget =: 3 : 0
'f t p'=. 3 {. (boxxopen y),a:
URL_httpget_ =: f
(2!:0) httpgetJS
RESPONSE_httpget_ (1!:2) <'file.txt'
0;'file.txt'
)

install_gitrepo_jpacman_ 'github:jpjacobs/types_dict'

For local installs, jsoftware.com also does not support CORs

 (0!:100) <'jlibrary/system/util/pacman.ijs' 

HASFILEACCESS=: 1
HASADDONSDIR=: 1

SystemFolders_j_  =: SystemFolders_j_ , ('install';'')

testaccess_jpacman_ =: 3 : 0
f=. <jpath'~install/testaccess.txt'
  '' 1!:2 f
  NB. 1!:55 f
  1
)

checkaddonsdir=: 3 : 0
d=. jpath '~addons/*'
if. # 1!:0 d do. 1 [ HASADDONSDIR_jpacman_=: 1 return. end.
)

install_addon=: 3 : 0
ndx=. ({."1 ZIPS) i. <y
NB. if. ndx = #ZIPS do. EMPTY return. end.
NB. log 'Downloading ',y,'...'
NB. f=. 3 pick ndx { ZIPS
NB. 'rc p'=. httpget WWW,'addons/',f
NB. if. rc do. return. end.
NB. log 'Installing ',y,'...'
NB. msg=. unzip p;jpath'~addons'
NB. ferase p
NB. if. 0>:fsize jpath'~addons/',y,'/manifest.ijs' do.
NB.   log 'Extraction failed: ',msg
NB.   info 'Extraction failed:',LF2,msg
NB.   return.
NB. end.
install_addins y
install_config y
)

NB. maybe we need to overwrite httpget?
httpget =: 3 : 0
'f t p'=. 3 {. (boxxopen y),a:
URL_httpget_ =: f
(2!:0) httpgetJS
RESPONSE_httpget_ (1!:2) <'file.txt'
0;'file.txt'
)

httpgetJS =: {{)n
    var url = jgetstr("URL_httpget_");
    url = url.replace("http://", "https://")
    var request = new XMLHttpRequest();
    request.open('GET', url, false);  
    request.send(null);
    console.log(url);
    if (request.status === 200) {
      console.log(request.responseText);
      jsetstr("RESPONSE_httpget_",request.responseText);
    }

}}

'install' jpkg 'convert/misc'

Possible workarounds

1. Host a CORS proxy that forwards a white list of URLs on

Other unknown challenges:

1. Addons use unzip and it's unclear whether unzip will work on wasm

[rdm]

For quite some time now, I have been wishing for J native support for the zip file format. (Technically plural -- file formats -- since zip supports several different compression schemes.) Henry Rich's efforts to speed up some scalar processing mechanisms are relevant towards achieving this. (There's arc/zlib but that still uses zlib which is native code.) Anyways that's in the future.

But perhaps wasm ought to bypass the zip mechanism entirely and instead rely on the ubiquity of http/https transport, for loading addons.

In other words, pull addons directly from github using github's raw url mechanism.

Every official addon has a github repo under the github jsoftware account. Every addon also has a manifest listing the relevant files. This should work for many addons. In other words, for 'math/calculus' load https://raw.githubusercontent.com/jsoftware/math_calculus/master/manifest.ijs and the parallel 'calculus.ijs'.

(The addons which require native support (like math/lapack2) are would not be initially usable here -- as a general rule, their native support would also need to be ported to wasm. In some cases, this would be a purely mechanical deployment issue and in other cases (like httpget) it would be more an issue of cutting out some unnecessary bulk.)

(Note that there's an issue here with release management. For an initial prototype it's probably fine to load the most recent copy from master. Eventually, we would need to be able to load stable releases, at least for addons undergoing active development. The simple implementation here would be to first try fetching the raw file with the release tag replacing 'master' and if that fails try again pulling from 'master'.)

[joebo]

In other words, pull addons directly from github using github's raw url mechanism.

I'm glad to see someone else thought of this. I was leaning in that direction too after thinking through the complexity of CORS and zip. I don't think it would be that bad to loop through the FILES in the manifest.js and download them. The downsides I see are:

1. We won't be able to reuse much of pacman.ijs on first take. We'd be implementing this from scratch in the Playground. I don't think it's a huge lift so long as we limit it to that initial prototype you mentioned.
2. Large addons (do they exist)? with multiple files may be slow since async downloads of multiple files will be tricky to coordinate since the whole operation needs to be synchronous back to J. I need to do some more research here. It sounds like an array of promises that need to block until they are all complete or some other async/await concept.

(The addons which require native support (like math/lapack2) are would not be initially usable here -- as a general rule, their native support would also need to be ported to wasm. In some cases, this would be a purely mechanical deployment issue and in other cases (like httpget) it would be more an issue of cutting out some unnecessary bulk.)

I was thinking the same thing. For an initial prototype, we'd leave out supporting addons with a binary dependency. Down the road we could have a addonxyx_wasm.zip as part of https://www.jsoftware.com/jal/j903/zips.txt (for example) but then we'd also need to sort out the unzip issue.

[rdm]

Yeah.. J assumes that load is synchronous. So that would be where you would have to resolve the promise made by our rebuilt pacman.

As for unzip -- note that zip can build an "uncompressed" zip file. And, it would be almost trivial to build a J implementation of zip which supports only this zip format. We could use that for bootstrapping unzip. (We might later enhance this with a few other compression formats. (I am imagining here a j unzip routine which first loads the central directory at the end of the zip file and if the "store" compress method was used unzips the file directly, otherwise it uses arc/zip. Conceptually, it would not extracting files, instead it would be returning the contents of a file, given an appropriate file name (or, conceptually, a directory list -- the zip file "owning" all contained paths.) There's a little bit of a trick here, structuring urls referring to zip file contents, but that should at least be straightforward. Sadly, though, I think browser javascript requires the whole file be "loaded' -- so we are at the mercy of the browser implementation for efficiency and viability in the context of "large files".)

Here's a quick implementation of "unzip uncompressed files using only native j": zip0,ijs gist

[joebo]

I've implemented a prototype of this in emj.ijs - https://github.com/jsoftware/j-playground/blob/j903_wasm/bin/html/emj.ijs#L488

Approach:

1. override httpget with a pure javascript implementation
2. override methods to download the manifest from github
3. execute the manifest to load the nouns FOLDER and FILES into the environment. I wouldn't normally do this on a desktop for security reasons but there is no harm on doing this in the browser sandbox
4. loop through each file and download using httpget

It works better than I expected for these example addons. More testing is needed - please test any addons and let me know if there are bugs

Here is a working example:

install 'github:jpjacobs/types_dict'
install 'github:AshyIsMe/jreference' 
require 'types/dict'
require 'ashyisme/jreference'

H

help_pdict_

Playground Link

[rdm]

Sweet.

Now to figure out how to stress test this... (my favorite addon stress test uses opengl support, but implementing webgl workalike support is I think still some time in the future)

[joebo]

Every official addon has a github repo under the github jsoftware account. Every addon also has a manifest listing the relevant files. This should work for many addons. In other words, for 'math/calculus' load https://raw.githubusercontent.com/jsoftware/math_calculus/master/manifest.ijs and the parallel 'calculus.ijs'.

This seems like a good path forward to use https://github.com/jsoftware/addonrepos/blob/master/repos.txt to install addons from git. For example

   install 'github:jsoftware/math_calculus'
installed: jsoftware/math_calculus master into folder: math/calculus
   require 'math/calculus'
   0 1 2&p. deriv_jcalculus_ 1
1 4&p.

Playground Link

I'm not yet clear how best to incoprorate https://github.com/jsoftware/addonrepos/blob/master/repos.txt into the playgorund. Maybe we implement our own instance of verb jpkg which search the repos.txt

For example

 'install' jpkg 'general/misc'

Would search the repos txt file and replace / with _ to find github:jsoftware/general_misc and then run install on i.

Potential design:

1. Add jpkg verb dyad that
2. Downloads repos.txt if it hasn't been downloaded yet for the session (instead of packing it with the playground, it's probably better to make it dynamic)
3. Searches the repos.txt for the y argument by replacing / with _
4. Runs installz on the match

[rdm]

Where does the playground store its files? (Are they anything more than browser cached references to the content fetched from the upstream urls?)

[joebo]

There are two types of files I would say.

1. Files embedded into the WASM at compile time (e.g plot addon and viewmat and emj.ijs [custom playground code])

2. Files dynamically downloaded at runtime using javascript XmlHttpRequest / J host foreign. These files are ephemeral and go away when the tab is closed

We could store add-ons in browser localStorage to cache them if we thought it was worth the effort

On Wed, May 25, 2022, 6:24 PM Raul Miller @.***> wrote:

Where does the playground store its files? (Are they anything more than browser cached references to the content fetched from the upstream urls?)

— Reply to this email directly, view it on GitHub https://github.com/jsoftware/j-playground/issues/64#issuecomment-1137901295, or unsubscribe https://github.com/notifications/unsubscribe-auth/AABFAFPW4T2F4J4JDOOPQZDVL2SDFANCNFSM5WXVP2MA . You are receiving this because you authored the thread.Message ID: @.***>

[rdm]

I was thinking in the other direction.

Typically, when I install J, I install all official packages. Basically install'all'

I was thinking about that approach in the context of the playground.

[bobtherriault]

If the addons are already in files, could you have the J playground access them locally?

Cheers, bob

On May 25, 2022, at 15:42, Raul Miller @.***> wrote:

I was thinking in the other direction.

Typically, when I install J, I install all official packages. Basically install'all'

I was thinking about that approach in the context of the playground.

— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you commented.

[joebo]

I'm not sure I understand the question so either I'm not being clear in my explanation or I am not tracking something.

This line in the makefile specifies what's included at compile time and bundled with the html/js/wasm file - https://github.com/jsoftware/j-playground/blob/j903_wasm/jsrc/makefile.html#L33

--preload-file ../jlibrary/ --preload-file ../labs/ --preload-file ../addons/ --embed-file ../bin/html/emj.ijs@emj.ijs

When I set this up initially, I started from a clean install and only installed plot and viewmat so that the dependency packages would be downloaded to addons. That's what's bundled by default.

I did not install all because I was worried it would bloat the file size (untested). That is a viable path to bundle all official addons if the file size wasn't too large. The assets bundled in the wasm are not compressed although that is something else we could tinker with if we wanted to shrink the size but would need to then solve the unzip issue previously mentioned.

Bundling as part of the build step is the equivalent of 'accessing locally'...

The code I wrote today downloads the addons and 'installs' it into the temporary filesystem that comes with wasm. https://github.com/jsoftware/j-playground/blob/j903_wasm/bin/html/emj.ijs#L504

It does this by interacting with XmlHttpRequest to download the files synchronously to the temporary session filesystem.

Does that help clear it up?

[rdm]

Actually, the simple answer is: yes.

Try this, for example:

   install 'github:jsoftware/math_calculus'
   fread getscripts_j_ 'math/calculus'

Or, in a fresh https://jsoftware.github.io/j-playground/bin/html2/ session:

   fread {:4!:3''

That said, J's 'open' verb (which loads a file into an edit window) is not yet supported in the playground. (And, it would be nice to have edit window tabs, in that context.)

[joebo]

Ah, I understand bob's question now and thanks for the reply raul. Some way to open the file in the playground would be an easy add. I will add an issue to track #72 to track it

On Wed, May 25, 2022, 7:11 PM Raul Miller @.***> wrote:

Actually, the simple answer is: yes.

Try this, for example:

install 'github:jsoftware/math_calculus' fread getscriptsj 'math/calculus'

That said, J's 'open' verb (which loads a file into an edit window) is not yet supported in the playground. (And, it would be nice to have edit window tabs, in that context.)

— Reply to this email directly, view it on GitHub https://github.com/jsoftware/j-playground/issues/64#issuecomment-1137934837, or unsubscribe https://github.com/notifications/unsubscribe-auth/AABFAFJBYHR7JCERGEJG3KDVL2XRPANCNFSM5WXVP2MA . You are receiving this because you authored the thread.Message ID: @.***>

[joebo]

Closing as completed. This facility is working well as far as I've seen. Let's open a new issue if anything new is needed with ability to install addons