Make the user aware of that they are using code from `experimental`

lkastner commented 1 year ago

I decided to merge #1999 and relocate the discussion on the separation of experimental from the rest of the source here.

We all seemed to agree that the user should in some way be made aware of that they are using code from experimental. Just for the sake of completeness: Please disagree here if you disagree.
Should code from experimental be available by default? As already stated, if not its discoverability goes down, but that is not necessarily a bad thing. In my pov code should not stay "experimental" for too long.

These were the ideas of @fingolfin :

Indeed. Some options come to mind:

* maybe accessing code in experimental should be "guarded" and require `using Oscar.Experimental`, or require writing `Experimantal.FOO` (so `Experimental.galois_group(...)`

  * pro: obviously experimental
  * con: less discoverable

* or maybe the primary functions should have a `@warn_experimental()` call in them, which prints a warning about them being experimental (possibly only the first time they are called)

* ... other ideas?

So I'll cc some people who might be interested: @micjoswig @fieker @benlorenz . Maybe we can discuss a bit here and then decide during one of the next Friday meetings, preferably when we are all there, unless we immediately find a consensus here.

fieker commented 1 year ago

On Fri, Mar 03, 2023 at 08:36:09AM -0800, Lars Kastner wrote:

I decided to merge #1999 and relocate the discussion on the separation of experimental from the rest of the source here.

We all seemed to agree that the user should in some way be made aware of that they are using code from experimental. Just for the sake of completeness: Please disagree here if you disagree. I do not agree here - but do not strongly disagree

Should code from experimental be available by default? As already stated, if not its discoverability goes down, but that is not necessarily a bad thing. In my pov code should not stay "experimental" for too long.

yep

Question: What do we want to achieve/ avoid?

lower the barrier to share code
make it available immediately
do not promise no further (interface) changes
???
s.o. getting wrong results (yes, we want to avoid that!!)

So if I write a function CoolStuff in experimental (OK, I will write cool_stuff), and expoert it, then

anyone can play with it
there is a slight danger that the code will immediately be used programmatically which might break later This would also indicate a more experienced user who might be able to discover the experimental nature alone
there is a slightly higher chance that s.o. will just interactively play with it - in which the discoverability is important and the breaking of th API irrelevant
there are/ will be stuff in experimental where the API is rock-solid (e.g. any grobener_basis function will return a groebner basis, any factorisation will return a factorisation, ...) here the experimental part indicated that the algorithm might be improved/ changed - based on feedback, or hooked up automatically or not or ...

How would anyone "discover" or use stuff in experimental - unless its automatically there? If they are on a mailing list, there might be an announcemet: cool_stuff available in Oscar now - and hopefully they would read further to disciver that they have to actively allow it first. My experience: they will just call cool_stuff and then be annoyed since it was not there and hopefull go back to read more.

So: what are the goals/ what are the pitfalls?

Claus

These were the ideas of @fingolfin :
Indeed. Some options come to mind:
* maybe accessing code in experimental should be "guarded" and require `using Oscar.Experimental`, or require writing `Experimantal.FOO` (so `Experimental.galois_group(...)`

  * pro: obviously experimental
  * con: less discoverable

* or maybe the primary functions should have a ***@***.***_experimental()` call in them, which prints a warning about them being experimental (possibly only the first time they are called)

* ... other ideas?
So I'll cc some people who might be interested: @micjoswig @fieker @benlorenz . Maybe we can discuss a bit here and then decide during one of the next Friday meetings, preferably when we are all there, unless we immediately find a consensus here.

-- Reply to this email directly or view it on GitHub: https://github.com/oscar-system/Oscar.jl/issues/2011 You are receiving this because you were mentioned.

Message ID: @.***>

HereAround commented 1 year ago

In response to Claus' question: "What are the goal?"

Maybe the intention is to avoid wrong results?

If so, I like the idea from a user perspective. Either having a warning (or having to invoke using Oscar.experimental) would then alert me that the results that I am computing might not be accurate. So the computed results must be used with care.

In case we agree to go ahead with this, I like both the warnings and using Oscar.experimental. The latter is easier to maintain, while the former might be better to inform that some results were computed with experimental code (hence, whenever one sees an "experimental code was used"-warning, the following result and results computed based on this must be used with care), while the methods in src/ would not trigger such a warning.

simonbrandhorst commented 1 year ago

From the user perspective I prefer the @warn_experimental() solution. But putting @warn_experimental() in each function seems annoying. Are there ways to automate it?

fieker commented 1 year ago

On Wed, Mar 15, 2023 at 06:47:53AM -0700, simonbrandhorst wrote:

From the user perspective I prefer the @._experimental()` solution. But putting @._experimental()` in each function seems annoying. Are there ways to automate it?

But why? So me, code in experimental is expected to work - and "never" return wrong results. So why put warnings in place? Careful: the next time it might be faster? The output (format) might change? Philosohically, I can understand the problems, but is this really a problem or a hypothetical one?

-- Reply to this email directly or view it on GitHub: https://github.com/oscar-system/Oscar.jl/issues/2011#issuecomment-1470041724 You are receiving this because you were mentioned.

Message ID: @.***>

simonbrandhorst commented 1 year ago

I would be hesitant to use experimental code to prove a theorem unless I wrote the code myself and know exactly what I am doing.

fieker commented 1 year ago

On Wed, Mar 15, 2023 at 09:03:53AM -0700, simonbrandhorst wrote:

I would be hesitant to use experimental code to prove a theorem unless I wrote the code myself and know exactly what I am doing.

Than we need to document our intend better. To me, experimental is supposed to be correct - as normal code, however, not in oits final shape and with the wrong method, wrong API, but mathematically correct. Code that is not correct, maybe should be in example or in a draft PR, but my intentions is that correctness is guaranteed

-- Reply to this email directly or view it on GitHub: https://github.com/oscar-system/Oscar.jl/issues/2011#issuecomment-1470314257 You are receiving this because you were mentioned.

Message ID: @.***>

simonbrandhorst commented 1 year ago

Where would you put code for a n algorithmic paper that is submitted but not yet published?

micjoswig commented 1 year ago

What is your goal? Do you want to keep that code secret? What is special about that code?

fieker commented 1 year ago

On Wed, Mar 15, 2023 at 11:37:07PM -0700, Michael Joswig wrote:

What is your goal? Do you want to keep that code secret? What is special about that code?

Why is it secret? It is as easy to find as everything else... Given that we are spread over >= 5 repositories, browsing a single directory will never show everything. Given that we also make use of sub directories and that many functions do not have a "natural" place (e.g. operation on ideals: part of groups or of ideals?, conversions (Ring -> additive group, ...) this is also unavoidable.

I find key new features mostly using word-of-mmouth and tab-expansion. Code usually by "methods", @.***" or "git grep". Eventually possibly in the "News" section of the "Release Notes" - once we have them...

As I said before: the goal to me is make imperfect (but correct! (up to the usual numner of errors)) available as soon as possible, while indicating to programmers that the APIs and such may be still subject to change. Futhermore, I'd like it as easy and painless as possible for all parties involved. I got annoyed (Magma, Maple) too frequently when trying some announced new feature to learn that it is only available after some secret activation (usually mentioned in the original posting, but usually also lost in the trickling down to me)

Annotating every (exported) function with a warning manually will just result in people not doig this. If you can do it automatically, then maybe.

What is the problem with experimental? Bad name? Wong concept? Abuse?

-- Reply to this email directly or view it on GitHub: https://github.com/oscar-system/Oscar.jl/issues/2011#issuecomment-1471399555 You are receiving this because you were mentioned.

Message ID: @.***>

thofma commented 1 year ago

If there is no difference between using code from experimental to normal code, there is no point of having experimental.

Either the code in experimental is different for the user (unstable API, maybe wrong results), then the user should be made aware of this explicitly. Or it is not, then this whole discussion is moot and experimental should just be removed.

micjoswig commented 1 year ago

What is your goal? Do you want to keep that code secret? What is special about that code?

That was meant as questions for @simonbrandhorst

simonbrandhorst commented 1 year ago

@micjoswig One of my goals is to put my code somewhere other people can use it [at least the parts that are of general interest] and where I myself can use it in 3 years. Because if I keep it private it starts to rot.
Let's say if one of us convinced that the code is correct this is good enough for us. But what if someone external believes their code is correct and wants to submit to experimental although his algorithm is only in preprint form. Probably we do not have the time to review it in detail. Should we reject that? Put it in examples? Experimental? src?j

fieker commented 1 year ago

On Thu, Mar 16, 2023 at 12:27:57PM -0700, simonbrandhorst wrote:

@micjoswig One of my goals is to put my code somewhere other people can use it [at least the parts that are of general interest] and where I myself can use it in 3 years. Because if I keep it private it starts to rot.
Let's say if one of us convinced that the code is correct this is good enough for us. But what if someone external believes their code is correct and wants to submit to experimental although his algorithm is only in preprint form. Probably we do not have the time to review it in detail. Should we reject that? Put it in examples? Experimental? src?j

That cannot be decided in general - this will, like in GAP I assume involve a reviewing process. Similar to papers, here the author is trying to convince us it is correct - and there will be few formal rules about the process.... -- Reply to this email directly or view it on GitHub: https://github.com/oscar-system/Oscar.jl/issues/2011#issuecomment-1472623325 You are receiving this because you were mentioned.

Message ID: @.***>

fieker commented 1 year ago

On Thu, Mar 16, 2023 at 12:27:57PM -0700, simonbrandhorst wrote:

@micjoswig One of my goals is to put my code somewhere other people can use it [at least the parts that are of general interest] and where I myself can use it in 3 years. Because if I keep it private it starts to rot.
Let's say if one of us convinced that the code is correct this is good enough for us. But what if someone external believes their code is correct and wants to submit to experimental although his algorithm is only in preprint form. Probably we do not have the time to review it in detail. Should we reject that? Put it in examples? Experimental? src?j

... but even then: unless we are convinced it is correct (up the usual...), it is geing to be at most in examples... -- Reply to this email directly or view it on GitHub: https://github.com/oscar-system/Oscar.jl/issues/2011#issuecomment-1472623325 You are receiving this because you were mentioned.

Message ID: @.***>

simonbrandhorst commented 1 year ago

I am okay with code in experimental to be "correct" so that the only basic difference is an unstable api. But right now when I code, I don't really know myself which part of the schemes is experimental and which in src. Perhaps it will help when we have all our exports accessible via grep?But it seems that @which experimental_foo returns the path and solves this issue too.
So I start seeing your point @fieker .

fieker commented 1 year ago

On Thu, Mar 16, 2023 at 12:18:32AM -0700, Tommy Hofmann wrote:

If there is no difference between using code from experimental to normal code, there is no point of having experimental.

Either the code in experimental is different for the user (unstable API, maybe wrong results), then the user should be made aware of this explicitly. Or it is not, then this whole discussion is mood and experimental should just be removed. For the interactive user, there is no difference at all For the programming user, they will read the code and know about it

For the contributer, I still maintain the psychological hurdle is lower to have experimental....

-- Reply to this email directly or view it on GitHub: https://github.com/oscar-system/Oscar.jl/issues/2011#issuecomment-1471435268 You are receiving this because you were mentioned.

Message ID: @.***>

fingolfin commented 1 year ago

A bit late to the game, but here is my view:

code in experimental should strive to be correct like all our code (but of course bugs happen)
but it may be incomplete (error("this case is not yet implemented")), unpolished, API not yet settled, etc.
thus I am fine with people using it, but I want them to be aware there are sharp edges and also that they might have to adapt to API changes in the future
in fact code in experimental also might go away again instead of graduating to src
of course right now, we are at a 0.x release of OSCAR anyway, so strictly speaking all our APIs can still change, so this is more a question of which degree of changes to expect
I am not sure if all programming users will read the code, in fact I doubt it (and honestly, I find that OK -- I read a lot of code but I cannot always read all of it!)
for me the goal of experimental is indeed to lower the entrance barrier for code

One can indeed use @which to find out if a function comes from experimental, but one has to (a) know about this, and (b) even begin to guess that this might be relevant -- if I find galois_group, I then read its docstring, determine it does what I need, and happily use it. Why would I even consider to think it might be experimental??

Here is a suggestion what we could do to move forward:

maybe the manual could have a section "Experimental" which explains what the meaning of experimental is, and how to determine code is experimental (the @which trick)
in addition, maybe doc strings for code in experimental could mention prominently a sentence like "This function is part of the OSCAR experimental code. It is perfectly fine to use, but please read REF to fully understand what this means"
we could even write our own drop-in variant of the md_str macro, which automatically detects when it is used inside experimental and then adds that verbiage to the end of the docstring
perhaps "experimental" is a name that still sounds too "risky" to some people. I.e., for some the connotation may indeed suggest that the code is unreliable. Perhaps we can find a better name. How about "staging"? Or "develop" / "development" ; or ...?

fingolfin commented 1 year ago

One minor point I forgot: perhaps each experiment could have an (optional!) README.md that says what the experiment is about and what its status is

joschmitt commented 7 months ago

I would like to open up this discussion again so we don't have to do it all on Friday. From what I can see, most people in this thread agree that the code in experimental may have an unstable API (and I think this is basically the defining property of experimental since who would consciously write wrong code?). But having an unstable API breaks the semantic versioning business. So, how do we proceed after 1.0? Let me throw the following options in the discussion:

experimental has to follow the Semver guidelines, so no API changes till 2.0. This would essentially mean to drop experimental completely from my point of view -- every contributor would have to start their own little package until the API is stable.
experimental is officially declared "unstable" and we somehow notify users by one means or another from the list above https://github.com/oscar-system/Oscar.jl/issues/2011#issuecomment-1485043175
[NEW!] We start a separate package with OSCAR as a dependency in which we put all experimental code. This package could then keep a 0.* version number forever and whenever some code is stable enough it can move to "OSCAR stable". This would of course require that currently no code in src depends on experimental (but in the long run I would declare this a "feature" enforcing our guidelines).

Pinging @lgoettgens @HechtiDerLachs @HereAround @StevellM since they seem to be the biggest contributors to experimental from what I can see.

fieker commented 7 months ago

will not change. Experimental is outside semver

thofma commented 7 months ago

I think everyone is happy that experimental exists and that it does not have to follow semver. This answers https://github.com/oscar-system/Oscar.jl/issues/2011#issuecomment-1940901930.

But it is still not clear, if one should be made aware that this can break at any time. I am mainly thinking about the code of people we don't know about. They happily use galois_group or your other favourite experimental function in their scripts and suddenly it does not work anymore, because someone added or removed another return value (or we removed the function altogether, because why not?). And no, they did not read the online documentation or called @which. They were looking for Galois groups and found galois_group.

micjoswig commented 7 months ago

We are currently looking into possibilities to use Documenter to mark experimental functions in the generated docs. Stay tuned.

lgoettgens commented 4 months ago

We are currently looking into possibilities to use Documenter to mark experimental functions in the generated docs. Stay tuned.

Triage sees this as the only reasonable way. Is there any update @micjoswig ?

micjoswig commented 4 months ago

@lkastner is this doable with the recent fixes in Documenter?

lkastner commented 4 months ago

The Documenter fix is unrelated. For this to happen we need to tap into the documentation which is collected by Julia itself. And there is no clean way to do it if we want it automatized, which we definitely want, since otherwise authors in experimental will just ignore this part.

oscar-system / Oscar.jl

Make the user aware of that they are using code from `experimental` #2011