Closed bvandenbroek closed 1 year ago
@bvandenbroek Thanks for the update. I compared this against https://imagej.net/ij/developer/macro/functions.html, which is where I would expect to find the latest macro functions documentation, and there are a lot of differences. It does not look clear-cut to me which one is "better" either: although the wsr.imagej.net copy has a much newer "last updated" date, it also introduces many HTML validity errors (e.g. removal of quotes), removes many "Requires 1.52o" type comments from various function docstrings, and even removes docstrings for some functions such as Color.wavelengthToColor
.
@rasband Could you please let us know which copy of functions.html is the canonical latest one? Then we can update the copy in this repository to reflect that, so that the Script Editor's autocompletion for ImageJ macros can function as smoothly as possible.
Hi @ctrueden , @bvandenbroek, @rasband, and also @haesleinhuepf
Sorry for chiming in a little late to this PR, I just realized this was being discussed.
First, some context for my intervention: I was already working on a branch to update several functions for the IJ macro autocompletion, as well as some of the code that deals with the parsing of the html to generate the suggestions. This started being discussed in this image.sc post. This update also incorporated some changes related to Fiji script editor update introduced by that time. The last update I pushed to this branch can be found here. At some point discussion in the forum pointed at some issues with the lexer, which made me postpone the PR to see if I could understand their inner workings (way over my skill level), and perhaps include a change in that same update. This was a clear error by my part. I ran out of time at the moment, put the project on hold, and here I am a little more than a year latter trying to catch up to finally release the original PR.
Anyways, going back to this particular PR, I believe that the functions.html
file in this library, which originally started as a copy of the website version, has slowly been diverging from that (irrespective of the addition of new functions), in order to accommodate for better parsing and entry disambiguation, and sometimes as a matter of style/html validity. One of the most important changes is the splitting of some anchor names (as in the case for close()
being split into "close", "close2", "close3", "close4") to allow for multiple entries in the contextual suggestions**. Some other examples are: adding closing </p>
tags that are missing, enclosing anchor names with quotes (and homogenizing to double quotes instead of single ones), remapping of internal anchors (like <a href="#IJ.renameResults">
) to the full url paths, replacing HTML symbols ( like <
for <
) etc. There are also minor edits and corrections (like adding a missing parameter to Roi.setGroup
).
As a result of all this, I have the impression that it is not currently viable to just copy the latest version of the file that is being maintained by @rasband for the web.
I think the function of both files is different enough to afford maintaining separate versions, although it is true that updating the one from the library implies some regex-replacing+diffing to track changes (this is what I have been doing). Other option would be to incorporate all these changes into the web version, and maybe writing a function to correctly translate internal anchors on the fly.
What are your thoughts on this?
Cheers, Nico
** in my branch I proposed a slight change, using "name_2" instead of "name2" to avoid a problem with atan2
The most up to date version of functions.html is at ` https://wsr.imagej.net/developer/macro/functions.html. I updated the version at https://imagej.net/ij/developer/macro/functions.html so it is a duplicate. I added unique anchor names ('close_pattern', 'close_all', 'close_others') for the various close() functions. I can incorporate other changes and fixes as needed.
@rasband Thanks for the quick reply
I gathered a (rather lengthy) list of changes that could be made to bring this two versions closer together. You can check the details below.
I feel like this is a lot of extra work that's unnecesarily falling upon you. That was the reason why I suggested keeping separate files, and offloading this burden to the ones contributing to/maintaining the autocompletion library. In this case, the html file in the library -for the lack of a better data format- could probably better be stripped of all the additional elements (headers, section links, letters, quote blocks, etc), so that it is clear that it's not just a copy of the web page.
However, if you still prefer updating the web file (either for use as a template to copy, or just for the sake of uniformity), is there a way to help you in doing this?
Also, if this is not the right place to discuss these issues, please let me know.
Thanks again.
@ndefrancesco @rasband I am in favor of one single source of truth for this information. Maybe the functions.html
could be committed to the https://github.com/imagej/ImageJ repository, such that it gets bundled into ij.jar
? It would be straightforward to make the pages served from https://imagej.net/ij/developer/macro/functions.html and https://wsr.imagej.net/developer/macro/functions.html stay in sync with this content via automation. And then here in imagej-legacy, the macro autocomplete could draw on the functions.html
embedded in ij.jar
rather than shipping its own. It would also allow the community to file pull requests to help with maintaining the content. What do you think?
@ctrueden that'd be a really good solution! Then it'd be easy to modify the function parsing the entries to correctly redirect the embedded links to the right URL, so that the file maintains the right format to be served on the website(s).
@rasband, @ctrueden
I've worked on a proposal to merge the most important changes that I had identified into the current version of the functions.html file that @rasband is maintaining. You can check it in this repo: https://github.com/ndefrancesco/ImageJ-functions-doc-update.
I started from the current version online at https://wsr.imagej.net/developer/macro/functions.html, and incrementally made the suggested modifications, plus some extra ones I had ready to incorporate. I tried to make the commits as specific as possible, so that it is easy to track what I've done. These changes are stucturally minimal, so that the way the page is rendered should not have mostly changed.
If you consider these changes are acceptable, this could be a good starting point to work towards @ctrueden's idea. This should at least save @rasband some editing time.
Please, let me know what you think.
Hi @ndefrancesco, @ctrueden,
The merged functions.html file looks good so I added it to the ImageJ repository at
https://github.com/imagej/ImageJ/blob/master/functions.html
The commit does not appear on the list at
https://github.com/imagej/ImageJ/commits/master/ij
Is this because no files were changed?
I did not modify build.xml to copy functions.html to ij.jar. I would prefer not to add this 190kb file to the ij.jar distributed with ImageJ.
Thanks, @rasband!
The commit does not appear on the list at
https://github.com/imagej/ImageJ/commits/master/ij
Is this because no files were changed?
It's because no files in the ij
directory were changed. You can see the commit if you change the URL to:
https://github.com/imagej/ImageJ/commits/master
i.e. drop the /ij
directory filter from the commit listing.
I did not modify build.xml to copy functions.html to ij.jar. I would prefer not to add this 190kb file to the ij.jar distributed with ImageJ.
That is unfortunate, but I understand the reluctance. I will figure out how to bundle it into imagej-legacy.jar at the linked version of net.imagej:ij
. Maybe the maven-remote-resources-plugin can do the job, I'll check.
Another option might be to download it on demand from https://imagej.net/ij/developer/macro/functions.html, although IIRC there was some performance reason that @haesleinhuepf did not originally want to do that... I guess because the network requests are too slow for autocomplete, and if we cache it then it can get out of sync as ij.jar evolves, so we would need to check at least once per JVM startup or daily or some such, whether the locally cached functions.html
is still as new as the one being served online.
Regardless, I will need to create some server-side infrastructure to make this committed functions.html
be the single source of truth aligned with https://wsr.imagej.net/developer/macro/functions.html and https://imagej.net/ij/developer/macro/functions.html. I can write a cron job that runs hourly and pulls the latest code from github.com/imagej/ImageJ.
@rasband What if I change https://imagej.net/ij/source/ not to be manually maintained, but rather just pulled from GitHub once an hour? Then I can symlink functions.html
from there to the other locations.
I decided to include functions.html with ij.jar. It makes things simpler and the file is not that large after compression. I updated Jerome's FunctionFinder tool (cmd-shift-f in ImageJ's macro editor) to use this file. I also removed all the 'Requires 1.52x' strings. "Requires" is only intended to be used with recently added functions.
@ctrueden: “What if I change https://imagej.net/ij/source/ not to be manually maintained, but rather just pulled from GitHub once an hour? Then I can symlink functions.html from there to the other locations.”
Sounds like a good plan. The files in that directory have not been updated in 7 months.
I have reconfigured https://imagej.net/ij/source/ to automatically pull from GitHub once an hour.
There was another copy of the sources at https://imagej.net/ij/download/tools/source/, which I have now symlinked to /ij/source/
.
I also reconfigured all existing copies of functions.html
to symlink to https://imagej.net/ij/source/functions.html, so that when a new version is pushed to source control, all the URLs will serve it after the hourly sync happens.
All previous copies of the source code, and previous copies of functions.html
, have been backed up to subfolders of https://wsr.imagej.net/backup/ as follows:
Old path | Moved to |
---|---|
/ij/source/ |
/backup/source/ij/ |
/ij/download/tools/source/ |
/backup/source/ij-download-tools/ |
/ij/source/functions.html |
/backup/functions/ij-source/functions.html |
/ij/developer/macro/functions.html |
/backup/functions/ij-developer-macro/functions.html |
/developer/macro/functions.html |
/backup/functions/developer-macro/functions.html |
/macros/functions.html |
/backup/functions/macros/functions.html |
/temp/functions.html |
/backup/functions/temp/functions.html |
/images.wsr.imagej.net/ |
/backup/images/wsr.imagej.net/ |
/images.mirror.imagej.net/ |
/backup/images/mirror.imagej.net/ |
These last two images
folders were relocated for consistency with the other backed up files, so that they don't clutter the toplevel of https://wsr.imagej.net/
@ctrueden I'm currently working on a new version of the IJ macro autocompletion to incorporate these changes. I've already managed to load functions.html
from ij.jar if available (currently only in the nigthly build). Would it be OK to also ship this updated version of the file with imagej-legacy.jar as a fallback in case ImageJ is not updated?
@ndefrancesco Oops, I was working on it also! I'll push the branch that I have.
@ndefrancesco OK, take a look at this: https://github.com/imagej/imagej-legacy/compare/functions-html-from-ij
I got it far enough along to compile, but have not tested it whatsoever yet. But hopefully it illustrates what I'm thinking? It avoids shipping functions.html
in imagej-legacy, in favor of reading it from ij, and falls back to reading it remotely if the version of ij.jar is too old.
If you have time to test it, I would appreciate it!
@ctrueden Thanks for the update!
I had some time to test that branch. It seems to work fine using the web version of the html file if the ij.jar version is missing, but it's not picking it up when it's present. I guess it should be easy to solve (it works ok in the previous implementation) but I'm not entirely sure what's going on with this new one.
There is, anyway, an aspect of that approach that I would prefer avoiding. It is true that if ImageJ is up to date (in this moment, beyond the latest stable release), there will be no need to fetch any file from an external resource. But in the case that imagej-legacy is updated without updating ImageJ (which can certainly happen during the transition, and will happen due to other circumstances), there will be an absolute need for a live internet connection for it to work, and as there's no caching of the file, that would happen every time it's launched. I personally dislike that option, as it's quite limiting.
On the other hand, the cost of having a (probably redundant) copy of the file [~48k compressed] will ensure that any version of ImageJ prior to 1.54g will have an up to date (and beyond) autocompletion, and any new version will be as updated as possible, with no need for connectivity in any case.
I have prepared a new branch where I gathered a bunch of commits from my previous attempt (with changes that have been already tested) and some new ones regarding this new file, including some logic needed to correctly parse this version (relative links, new function families). You can check it out here: https://github.com/ndefrancesco/imagej-legacy/tree/autocompletion_2023 I tried to make the commit descriptions as clear as possible.
In this case, I took the route of keeping a static copy of the file as fallback. Anyway, I think that the rest of the changes are fitting irrespective of what's the final decision regarding the way to handle a possibly missing functions.html
. There might be other issues that I could be missing that favor the first approach over the second.
Let me know what you think and how you suggest we move forward.
Thanks!
This pull request has been mentioned on Image.sc Forum. There might be relevant details there:
https://forum.image.sc/t/how-to-type-non-smart-quotation-marks-in-the-fiji-macro-editor/84909/3
Hi @ctrueden @ndefrancesco @rasband, Just wanted to thank you for picking this up so well! I'm happy to test things from the user perspective. :-)
Update for the IJ1 macro autocomplete functions in the Fiji editor. The page is copied from the new location: https://wsr.imagej.net/developer/macro/functions.html