Some sort of policy to regarding unexported functions in Base?

[IainNZ]

There have been a couple of times during 0.4 development where unexported functions have been removed or changed. Every time its come up we seem to go in circles a bit about what to do about it. Given the resources available, I feel we should just keep it simple:

No support or deprecation warnings, just change/remove them. Only exported functions get deprecation warnings or mitigation regarding changing arguments.

But we need consensus around this to make it stick.

[rsrock]

@ScottPJones Likewise, I have plenty of experience in the restrictive, Java and Java-like interfaces, and the more open, Python and Python-like systems. I greatly prefer the latter, especially for technical computing (the target audience of Julia).

Since most people seem to feel that by default, most things that are not exported should be treated as private, the simplest case would be to have a public keyword to indicate something (method or field) is part of the public API (which would only needed for non-exported names, a rare case).

But that's the problem-- I want the interface to be defined through export (+ docs), but to be able to access methods or fields, and I don't want the pile of sand to scold me about it. In other words, I want it to be somewhat troublesome to define something as private. I want you to have to stop and think "does this really, really need to be private?" The friction it introduces for privacy is a positive feature of this system.

In my reply to Stefan, I forgot to link the relevant section of the Python manual where they discuss privacy. I think they do it right, and they do have a rather long track-record to show that this _ and even __ business works pretty well.

https://docs.python.org/3/tutorial/classes.html?highlight=private#tut-private

[rsrock]

@phobon You can do this right now, with import vs. using:

julia> module SomeMethods
       foo() = 1
       bar() = 2
       export foo
       end
SomeMethods

julia> import SomeMethods

julia> foo()
ERROR: UndefVarError: foo not defined

julia> SomeMethods.foo()
1

julia> bar()
ERROR: UndefVarError: bar not defined

julia> SomeMethods.bar()
2

julia> using SomeMethods

julia> foo()
1

julia> bar()
ERROR: UndefVarError: bar not defined

julia> SomeMethods.bar()
2

[jdlangs]

@rsrock The intention was to let the package author annotate what methods are considered the public interface without also requiring them to make them visible via using. Anything not exported would then be liable to change without warning or deprecation.

Essentially combining @StefanKarpinski 's public keyword with the export statement.

[rsrock]

I see, to cover cases like Profile.print()? I suppose that export SomeMethods.foo could then be a no-op. That way, using SomeMethods will still require you to use SomeMethods.foo(). Interesting idea, that means that export could define the entire public interface after all.

[mbauman]

+1 to a export SomeModule: names. These names would show up in names(SomeModule), and tab completion could be restricted to names(SomeModule, #= all= =# false).

Also note that you can completely restrict access to local bindings with let. But I don't want to encourage such behavior in general. :)

[StefanKarpinski]

So export would not actually export anything? Confusing

[jdlangs]

Well, it wouldn't be a total no-op, since as @mbauman suggested, the module reflection methods would know that such identifiers are meant to be public. It would imply changing the meaning of export from the definition you gave though.

[StefanKarpinski]

My point is that then the term "export" would have lost all appropriateness and we might as well rethink what we call it. The business of using import to extend things is already pretty confusing.

[rsrock]

True. To add to this import doesn't import what's exported. Perhaps a rename would be in order.

[ScottPJones]

Seriously, let’s say that @StefanKarpinski ’s public keyword is added, and also a .! syntax to override it (and to address #1974, see @JeffBezanson s comment https://github.com/JuliaLang/julia/issues/1974#issuecomment-71538055), and the default is to act exactly as now, so that absolutely no change to current code is required, and a command-line switch is added for a "enforce accessibility" mode.

What would that mean? People who want everything to be public don't have to do anything at all, they wouldn't even notice the change.

People who really care about encapsulation, long term reliability, not accidentally breaking user code could use the public keyword.

We might want to require that packages that are registered be able to work with the "enforce accessibility" switch turned on (they can still use .! to get around that, but those can be trivially searched for if somebody wants to modify something that is non-public, and people using .! would know that it is their responsibility to fix things if things break.

Lint.jl could use the information to give warnings to people using non exported, non public names (without warning unnecessarily for things that are non exported but meant to be public).

Maybe a special public * syntax could be added also, to be used at the beginning of a module, to simply say that everything not exported is meant to be public.

[MithrandirMiles]

Another idea, get rid of export, and add public, where you can give the fully specified name, if you only want it accessible fully specified, such as people have suggested above, but without the jarring clash of export not really meaning what it does (as Stefan rightly said)

[quinnj]

Things to add to add to enhance namespace management:

• public keyword
• .! syntax
• command-line switch for "enforced accessibility"
• require reigstered packages to enforce accessibility
• public * syntax
• make export a no-op, but work for modules

Seems simple enough :stuck_out_tongue:

But seriously, lot's of good ideas/examples here. I think, given the breadth of the issue here, we may need to take a step back and examine a broader set of changes that wholistically address the issues here; I'm just not sure a syntax change or two will get us there.

[ScottPJones]

With a couple more months under my belt (and some more free time), I'd feel confident to submit a PR to do just that (although I didn't say to make export a no-op, rather to deprecate it and replace it with public, and add the capability to say public Foo.bar, as other people have suggested for export).

I think you are right, and also Stefan's comments about import, some serious thinking about how everything fits together seems necessary, not just isolated syntax changes here and there.

[mbauman]

Adding new reserved keywords is a multi-year project, so let's not blithely jump into that unless we're absolutely certain.

I do think tab completion is a great place to draw a line between public and private. It'd be wonderful if Profile.tabtab only listed the supported API for folks like me that can never remember what things are called, but that can also be done today with an Internals submodule. Heck, that might be the best approach for modules that intend to be used fully-qualified.

[StefanKarpinski]

@quinnj's point is not that we shouldn't do it because it's hard to implement but that it's a bad plan because it's a ton of features and complications. Which I agree with – too much stuff.

[ScottPJones]

I don't get that at all from what @quinnj said: he seems to be advocating for more changes, not less.

think, given the breadth of the issue here, we may need to take a step back and examine a broader set of changes that wholistically address the issues here.

I agree with that, I think it is time that instead of just patching over problems, and people complaining every now and then about import / export etc. being confusing, etc., thought needs to be put into handling a bunch of those issues in a consistent fashion, that people will be happy to live with for the rest of the life of the language.

[rsrock]

Following @quinnj (and now @ScottPJones, as I'm typing this) and thinking about this a bit more, I figured it was time to read the manual before suggesting any solutions. Here's what the docs say about each of these (paraphrasing). All of this is in the "Modules" chapter.

import: 1) control which names from other modules are visible. 2) operates on a single name at a time (?? not sure this is true). 3) allows functions to be extended with new methods.

export: specify which of your names are intended to be public.

using: module will be available for resolving names as needed.

Although I agreed with @StefanKarpinski on one of our proposed changes with export SomeMethods.foo(),

So export would not actually export anything? Confusing

...the actual definition of export seems to allow such use, in the sense of "Export a public interface". So we could stick with export instead of splitting hairs with public and export, in my opinion.

In Python, from foo import * pulls in everything without an underscore into the local namespace, and export / public is not even an option. So we're already more restrictive than they are.

As an aside, @StefanKarpinski also mentions

The business of using import to extend things is already pretty confusing.

Perhaps (at the cost of adding a keyword and code churn): extend? E.g. extend Base.getindex()

[mikewl]

Another question should be: public default or private default. So instead of explicitly marking the public interface, explicitly mark the private portions of the module or type.

As has been mentioned, there should also be a simple way to escape the encapsulation. If I want to do something with the internals I don't want to jump through hoops to please the microwaved sand gods. That being said, the way to escape encapsulation must be distinct enough to be noticeable when bug hunting.

[tknopp]

I also think that the submodule issue should be discussed in relation to the other module discussions (e.g. the import thingy Stefan mentioned)

The origin of this thread initialized by @IainNZ was, however, about plain (non-submodule) functions in Base and I fully agree that that these need no deprecations warnings. Non-exported public submodule functions are a different business and it seems to be appropriate to open a dedicated issue since this has been become to broad.

[tknopp]

Thanks @rsrock for suggestion reading the manual. It says

Within a module, you can control which names from other modules are visible (via importing), and specify which of your names are intended to be public (via exporting)

I think this sentence makes it pretty clear that export==public is the approved rule (with submodules forming an exception).

[ScottPJones]

@tknopp, the problem is, that simple black or white definition is simply not good enough, and doesn't match what people are actually doing, in base or in packages. Many times you don't want to pollute the namespace by exporting the name, but you do want the qualified name to be accessible, visible to help, etc. (and conversely, there are many times when you make it clear when some name is not meant to be public) Where I worked, breaking customers code was a big no-no, so it was very important to only make public things that you were sure you wouldn't be changing in the future. The constant breakage that goes on currently with julia might be OK for some people, but will need to stop soon as the language matures.

[tknopp]

The constant breakage that goes on currently with julia ...

Scott, your statement is totally unfair for the people that put a lot of effort into Julia: There is no constant breakage of the public interface (-> export, see the manual) of Julia. Additionally, the stable Julia version (0.3) is rock solid and the minor releases do not break any packages. The development branch is a different thing and the recommendation for the "customer" is to not use it.

[ScottPJones]

It's not an attack, it's just what I've seen since I've started with Julia in April. As I noticed at JuliaCon, many people need the features that are only available on 0.4, and hence have no other choice. Also, unless you want a permanent split between 0.3.x and 0.4, a la Python 2.x vs. Python 3.x, then at some point people will have to deal with the major changes between the two.

I'm not saying the changes are a bad thing at all, just that Julia could use some better mechanisms to help people isolate things that are meant to be public (but not necessarily exported, for name pollution reasons), and ones that really are internal, which should be stayed away from unless you want to risk high chances of future breakage.

[KristofferC]

Where I worked, breaking customers code was a big no-no, so it was very important to only make public things that you were sure you wouldn't be changing in the future.

Comparing the development of Julia to the development of some locked down commercial corporate software is in my view not very relevant. I believe most people working with Julia expects (and desire) rapid development with the trade off that things might have to break from time to time.

Also, just because 0.4 is even more awesome than 0.3 and people want to use it, doesn't mean that we can suddenly stop developing it. If you are on development branch then things might break. It is as simple as that. If you need something stable AND the features of 0.4 I guess you have to wait a couple of weeks.

[ScottPJones]

I believe most people working with Julia expects (and desire) rapid development with the trade off that things might have to break from time to time.

I heard quite a bit of grumbling at JuliaCon about breakages. I think the issue is really, what can be done to minimize breakage, while still allowing rapid development of the language. Compat.jl and the deprecations are good examples of things that do help that goal, and I think that being able to make a clear distinction between what the author believes to be part of the public API or not also would help that goal.

[mbauman]

This is getting all jumbled up with other issues.

I like Iain's initial suggestion — non-exported functions are officially not supported unless otherwise documented — with some of Tony's caveats about avoiding gratuitous breakages. As someone who has added a deprecation for a non-exported method, I'd add that I did so because in that case it was really easy… but I definitely don't want to set a precedent that all internal methods should be deprecated. If we notice that an internal function is getting used often, we should identify why and work on moving it towards a public API.

It'd be very interesting if we could search through packages and identify uses of internal functions. To make that work, though, we do need some mechanism for making a name public but not exported to support things like Profile.print(). Perhaps it's as simple as having documentation in the __META__ dict.

[tknopp]

I don't think this is enough. We need a way to address the submodule case in order to write a function public_interface(m::Module). Since internal functions also can have documentation it is too indirect to go through __META__.

[mbauman]

Since internal functions also can have documentation it is too indirect to go through __META__.

I agree that going through __META__ is a little indirect, but I think that, at least in Base Julia, only public functions should have docstrings. Internal methods can get comments in the code. I've been missing apropos since the Doc switchover, and a reasonable way for it to be re-implemented is to search through all the values of the __META__ dictionaries. If internal methods began showing up there it'd make the situation worse.

[ScottPJones]

I disagree strongly about internal methods not being able to have docstrings. I might have fairly complex internal methods, that I want to have real documentation, and annotate things like what exceptions they throw, etc.

[tknopp]

@mbauman: True. But these private docstrings could be accessible through some other functions. Thats why I think it would be better to have a direct way to define the interface of a module (i.e. export + some clever improvement to let Print.print go through some sort of export). If you have public_interface(m::Module) it could be used in apropos for filtering the actual API.

[ScottPJones]

@tknopp, I think private docstrings should be accessible from help, but maybe by doing something like doing two ?. About export, I agree with Stefan, that the name export doesn't make as much sense anymore, so I would advocate public as a preferred alias, i.e. public Print.print, foobar, blech, exactly as has been suggested for extending export.

[timholy]

@ScottPJones, I suspect you're in the minority about wanting internal methods to have docstrings. That's what code comments are for.

[ScottPJones]

When you are programming / debugging on a large project, you'd like to be able to have help come up, and not have to go digging into the source code all the time. Code comments simply don't cut it.

[toivoh]

I also think that allowing docstrings on internal functions would be nice, seems strange to go through the trouble of implementing nice documentation facilities and then restrict them so that they can't be used with all functions.

[timholy]

And we go around in circles again. There are two camps: people who want to make internal functions easier to use (so they should have docstrings), and people who want to make sure our API is so nice that there's never any need to call an internal function. I'm in the latter camp. The absence of docstrings for internal functions is a way of forcing us to get the API right.

Code comments are just fine for documenting something whose usage you actively want to discourage.

[tknopp]

Tim you are not alone. It would be pretty strange if our public API documentation would include documentation of internal functions. I don't care if there is some help_internal function though.

[ScottPJones]

@timholy You are forgetting the people who have to develop the software that uses those internal functions. I'm not trying to make internal functions easier to use outside of the module they are in, rather make it so it is clear they are internal, which doesn't happen now, but also, to make it easier for the people maintaining the package or module, or for the poor programmer that gets a backtrace with that internal function, who'd like to quickly find out what it is supposed to be doing. That's why I thought a ?? at the REPL, which could call a help_internal, would be nice.

[mbauman]

We have @less, @edit, etc. We're going down yet another rabbit hole here. I only brought up docstrings because they seem to be an easy and straightforward way to define the public interface by convention right now, without adding any new language features.

[StefanKarpinski]

This discussion is getting nowhere quickly and isn't going to need to be settled for several months. How about we all let this sink in and simmer for a bit and revisit the issue when it's more pertinent.

[rsrock]

@StefanKarpinski I just finished some doc edits (two minutes ago) that attempt to lay out how things currently work. It could either help frame the discussion, or it could get the pot boiling again.

Shall I hold off on the PR?

[StefanKarpinski]

No, go for it – improvements to the documentation of how things currently work are always good.

[rsrock]

Ok, it's in #12696

[aviks]

That is a good advice @StefanKarpinski . But allow me to make a meta point

The issues raised here go the heart of what kind of a language Julia is. Is it a permissive language or a restrictive one. Is it a language that makes many weird things possible, or a language that prevents many bad things happening...etc.. etc... (There is no value judgement in either of these, nor are they binary.. but decisions like this go a long way to define the feel of a language)

For things like this, I think any movement towards "design-by-committee" are dangerous. Which does not prevent me from expressing an opinion, not should it, for anybody. I'm sure they all are useful input. But we should all be wary of all our wishes coming true.

[tknopp]

@aviks: These are absolutely great points and if one follows the issue tracker and the mailing list you can see that the core maintainers see Julia as a permissive language. One issue is that there are sometimes newcomers having a different background and a very strong opinion which is expressed in verbose form so that one has to read these issues carefully to understand which is the "spirit" of Julia.

[ScottPJones]

@tknopp One can still have the language be permissive, and at the same time add support for programmers to express their intentions, and help make things more reliable. As @aviks said, it is not a binary issue - the language can be both permissive, and yet have mechanisms that be used to help prevent bad things from happening.

[tpapp]

In the meantime, until this issue is fixed, maybe it would make sense to standardize on some sort of style convention for functions which are not part of an API, which could even go in the style guide.

A suggestion from existing practice: many libraries use a _ prefix for "internal" functions, and for the packages I have installed,

$ grep -ro 'function _' ~/.julia/v0.4 | wc -l
752

If eg .! or some similar syntax is introduced, uses of SomeModule._hey_you_shouldnt_use_this() can easily be found and replaced. If this issue is never resolved, at least we have a convention that is orthogonal to docstrings and other issues.

[IainNZ]

Tidying up old issues/PRs.

[tpapp]

@IainNZ: how was this issue resolved?

[IainNZ]

I believe indirectly its been resolved by there being a 1.0 soon, which has (or will have) guarantees. There wasn't much value to be had from this years-old discussion.