Xonsh needs a way to run startup scripts after load time

[eode]

Currently, .xonshrc and files in $XONSHRC are loaded before xonsh variables are present. There was a discussion about this, the main takeaways being:

• Implementing a $XONSHRC_POSTLOAD would be the preferred method
  • this would be a list of files (and functions?)
  • this would probably have a default or two, like ['~/.xonshrc_postload']
  • missing files would be silently ignored, as with $XONSHRC

Other Ideas tossed around:

• Use a function or comment token to split the file into two sections. Execute the first pre-load, and the second once the shell is loaded. This doesn't really have any drawbacks that I see offhand.
• cleaning up the startup process so perhaps this isn't necessary might work, but would take no small effort
• Having a function or group of functions, like postload_foo() which would be called if created in .xonshrc
• Having a second .xonshrc file without envvar

[scopatz]

Thanks for opening this!

[wshanks]

Just to add another variation, in #1036 I suggested using a function (or just a special comment) with a name like wait_for_shell that could be put in .xonshrc. At startup, xonsh would split the file and use the part before this statement as it does now. Then after the shell is fully loaded it would execute the part of the file after this statement. This would just avoid having to have multiple files.

[eode]

Huh. I didn't notice your comment about having a special comment that splits the file in two.. ..that's actually not a bad idea. Each file in $XONSHRC is loaded up to the special function call / comment, and then each file is continued after startup. Not a bad thought at all.

[scopatz]

Of the listed options, I am pretty much only in favor of a separate environment variable.

[melund]

I also think an environment variable is the best and simpler option.

Another possibility (maybe more suited for xontrib) is hook functions. Pytest has a cool system for that from which we may get inspired. https://pytest.org/latest/writing_plugins.html

[eode]

nod I think that $XONSHRC_POSTLOAD is easily the simplest option to implement, its only drawback being that more than one config file would be required in some setups. I don't think splitting on a token / function call is exactly pretty (implementation side) but I do think it has merit, in that it's nicer on the usage side. Fundamentally, though, I have no opinion on the matter. It's either a bit of a mess on implementation or a bit of a mess on usage.

[scopatz]

I think that $XONSHRC_POSTLOAD is easily the simplest option to implement, its only drawback being that more than one config file would be required in some setups.

Yeah, and I would note that we already have that possibility for some configurations since $XONSHRC is a list.

I don't think splitting on a token / function call is exactly pretty (implementation side) but I do think it has merit, in that it's nicer on the usage side.

I actually disagree here. I have seen this with programs like vistrails and it is a total mess. It also forced users to learn a special API. Under the envvar method, all xonshrc file are potentially the same and one simply chooses when to load them. No special xonshrc API needed.

Fundamentally, though, I have no opinion on the matter.

Then I think we have a decision as long as there are no major objections :)

[wshanks]

Would $XONSHRC_POSTLOAD allow function names as well as files? One could then keep the post-load code in the same .xonshrc file inside a function that is put into $XONSHRC_POSTLOAD.

Also, does anyone have a list of reasons why .xonshrc can't be run after the shell is loaded? Is it solely the shell backend choice that prevents this?

[scopatz]

Would $XONSHRC_POSTLOAD allow function names as well as files? One could then keep the post-load code in the same .xonshrc file inside a function that is put into $XONSHRC_POSTLOAD.

I was planning on it just being file names.

Also, does anyone have a list of reasons why .xonshrc can't be run after the shell is loaded? Is it solely the shell backend choice that prevents this?

At this point, probably yes. I guess we could force people to set this in the config.json file instead... In general, xonsh startup needs to be a bit more streamlined. It has just sort of evolved based on need.

[wshanks]

Hmm, setting aside the issue of backwards compatibility, moving the sourcing of .xonshrc to after the shell is loaded and forcing people to choose the shell in config.json seems like the cleanest approach. That way, .xonshrc has access to the full shell as one does in other shells. Also, then one only needs two files (config.json and .xonshrc) and not a third $XONSHRC_POSTLOAD file.

Breaking backwards compatibility might not be too bad since the default shell type is best and it is likely that not too many people have prompt-toolkit installed but want to use readline. I haven't looked closely at how environment variables work but maybe $SHELL_TYPE could be made into a property that emits a warning if the user tries to set it.

[scopatz]

Yeah this might be the best option overall. I am not sure how much I like the not making shell type writable or issuing a warning. It seems like there are legitimate use cases for setting the shell type (such as when opening subshells). However, there may be other cases where read-only variables would be desirable. That is probably a bit beyond this discussion though :)

[adqm]

I agree that none of the options enumerated so far is particularly appealing.

Hmm, setting aside the issue of backwards compatibility, moving the sourcing of .xonshrc to after the shell is loaded and forcing people to choose the shell in config.json seems like the cleanest approach.

The problem for me is that with this approach, one must have two config files if they want to choose their shell (i.e., I would have to have two config files instead of one, whereas I only need one in the current setup).

In my ideal world, we would:

• get rid of config.json
• move .xonshrc execution earlier in the process
• add a .xonshrc_postload or similar

Necessary in this would be removing the option of customizing the .xonshrc location (just the normal local and global locations), though the location(s) of the .xonshrc_postload file(s) could still be customized.

What we lose is a config file that is easily machine-writeable. But we gain something, too; specifically, I would argue that having only one config language makes xonsh easier to learn and use.

A different approach would be to add several hooks throughout xonsh, where user-defined functions can be executed. There have already been a couple requested:

• after loading the shell
• before displaying the prompt
• before executing a command

If we took that approach, we could use the same mechanism for all of these cases, which has its perks. The downside, of course, is that these functions wouldn't be executed directly in the environment we're actually interested in changing, so actually modifying relevant values might be a bit obnoxious (though maybe we could specify these as blocks instead of functions and create a way to execute those blocks directly in the parent environment (shouldn't be too hard).

[gforsyth]

The problem for me is that with this approach, one must have two config files if they want to choose their shell (i.e., I would have to have two config files instead of one, whereas I only need one in the current setup).

This is a very good point especially with regard to usability / ease of adoption. More action should be required on the part of the user if they want to enable advanced features, rather than requiring more setup of beginners.

[scopatz]

Alternatively, we could have a way of allowing XONSHRC to be, rather than a tuple of strings, a tuple of tuples of string, where the first element is the file element and the second element is the a flag for what stage it should be loaded on. This would allow all types of files (xsh, json, py, etc) to be loadable at each stage, and maybe would could transition to dropping json support. something like:

$XONSHRC = (
  ('config.json', 'preshell'),
  ('xonshrc', 'postshell'),
  ('post.xsh', 'postload')
)

[PaulReiber]

Adam wrote:

A different approach would be to add several hooks throughout xonsh, where user-defined functions can be executed.

• after loading the shell
• before displaying the prompt
• before executing a command

+1 for working toward that sort of solution. Additionally, we could integrate into this a fourth bullet item, "after changing directory" and that'd implement the idea Anthony and I've been discussing in another thread. I've been thinking on how to implement cd-actions in xonsh, and had been reading thru the code hoping to find exactly this - examples of other hooks. Maybe we can implement some common infrastructure within xonsh that all of the above can leverage.

Anthony wrote:

a tuple of tuples of string, where the first element is the file element and the second element is the a flag for what stage it should be loaded on ... sure, that could work too - maybe we could extend "stage" so it includes some new states preprompt preexecute and postchdir?

Xonsh walks an interesting path, keeping Python on one side and BASH-ishness on the other. For per-user customization, I'm all for getting it to be more Pythonic and less BASH-ish. I think that's where its power can shine.

On the other hand, part of me thinks commandlines are "pipelines" by default, and almost never intentionally "or cases" typed in manually. By extension the first | on a commandline would need to be examined to see if what's after, to EOL, is valid in subprocess mode, and if so, auto-wrap the left-hand-side python code in @() so that pipe isn't accidentally interpreted as an "or". Because, who actually intentionally uses "or" on a commandline, vs. wanting to pipe the output. So, for that case, I think BASH-ishness would be a better default. Maybe a setting "OR_MEANS_PIPE" to let the end-user decide?

Thoughts?

Kind regards,

_Paul Reiber_http://about.me/reiber

On Sat, Jun 25, 2016 at 9:44 PM, Anthony Scopatz notifications@github.com wrote:

Alternatively, we could have a way of allowing XONSHRC to be, rather than a tuple of strings, a tuple of tuples of string, where the first element is the file element and the second element is the a flag for what stage it should be loaded on. This would allow all types of files (xsh, json, py, etc) to be loadable at each stage, and maybe would could transition to dropping json support. something like:

$XONSHRC = ( ('config.json', 'preshell'), ('xonshrc', 'postshell'), ('post.xsh', 'postload') )

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/xonsh/xonsh/issues/1204#issuecomment-228584092, or mute the thread https://github.com/notifications/unsubscribe/AADlQLc4DjEATGu5MzwL_n3pJ9KCMgCgks5qPgOzgaJpZM4IzUsQ .

[melund]

A different approach would be to add several hooks throughout xonsh,

I also like this options the most. A set of hooks functions would allow the .xonshrc or xontribs to really modify the behaviour of xonsh.

I think pytest is good example of this: https://pytest.org/latest/writing_plugins.html#writing-hook-functions

[scopatz]

I am not a huge fan of the pytest architecture. It makes users learn an API, which is fine if those users are also developers. However, for xonsh (and vistrails and other non-developer facing tools) I think forcing this on new users would be too much.

That said, I think that most of the functionality that one might want out of such a hook system could be gained by simply by allowing callables as well as file names in the list-of-tuples strategy. I would definitely be in favor of this.

As a benefit, this would also allow API migrations or difference because the flag/stage could be different depending on which API the function should use. Unless we would go with **kwarg only for the API, this is a big concern of mine with the function hook idea.

[scopatz]

On the other hand, part of me thinks commandlines are "pipelines" by default, and almost never intentionally "or cases" typed in manually. By extension the first | on a commandline would need to be examined to see if what's after, to EOL, is valid in subprocess mode, and if so, auto-wrap the left-hand-side python code in @() so that pipe isn't accidentally interpreted as an "or". Because, who actually intentionally uses "or" on a commandline, vs. wanting to pipe the output. So, for that case, I think BASH-ishness would be a better default. Maybe a setting "OR_MEANS_PIPE" to let the end-user decide?

I don't understand this @PaulReiber. Can you explain it again, please?

[scopatz]

Also, I suppose that we could have an API too, if all that the API did was modify the underlying model.

[wshanks]

The problem for me is that with this approach, one must have two config files if they want to choose their shell (i.e., I would have to have two config files instead of one, whereas I only need one in the current setup).

This is a very good point especially with regard to usability / ease of adoption. More action should be required on the part of the user if they want to enable advanced features, rather than requiring more setup of beginners.

I wonder what background can be assumed for the typical xonsh beginner? Familiarity with bash? As a beginner, I found it surprising that xonshrc was not sourced in a fully loaded shell (if xonsh ends up just adding a post-load file, I'll just move my xonshrc to the post-load file and my xonshrc will just set the post-load file). Also, I would guess that the number of beginners with prompt-toolkit installed who want to use readline is pretty small (unless prompt-toolkit is bundled into the xonsh package for some systems?).

I am not sure how much I like the not making shell type writable or issuing a warning. It seems like there are legitimate use cases for setting the shell type (such as when opening subshells).

Here is an alternative suggestion: $SHELL_TYPE could be made into a property that loads the shell the first time it is set in xonshrc (or after xonshrc if it's not set). This would let you set the shell type wherever you wanted in xonshrc and execute commands before and after it is loaded. I haven't studied the startup process. Maybe there are other side effects to the loading process that would come into play with this.

In general, I like the idea of hooks, though other than needing a post-load one for the way things currently work, I can't think of an event I would need to hook onto (directory change would be nice but you can do this now by adding a function to the prompt FORMATTER_DICT that checks the directory).

I like the idea of consolidating the start up files. Is there anything that config.json can do now that couldn't be done in xonshrc?

[PaulReiber]

I don't understand this @PaulReiber https://github.com/PaulReiber. Can you explain it again, please?

I'll try, sure. What's it mean to be "at the command line?" Here's my proposal, regarding xonsh in interactive mode.

I guess it's all a matter of perspective - when you're typing at the command line, what are you trying to do? From sh, csh, tcsh, zsh, bash, korn shell onward the command line has been a means to let one build useful pipelines on the fly. Processing cascades - outputs of one process being sent to the next, then the next, then the next, as needed.

So, for xonsh, sure, Python's primary, but when in interactive mode, when encountering a | what does that mean?

I use interactive shells to construct pipelines, not or-cascades. I love the idea of piping the output of a bunch of relevant python code to some subprocesses by simply "piping it":

$ 4 + 4 | cat 8 $

It's the one place where I think being non-pythonic might make some sense. And, as I said, it could be left to the end-user to decide.

Kind regards,

_Paul Reiber_http://about.me/reiber

On Sun, Jun 26, 2016 at 7:40 PM, Anthony Scopatz notifications@github.com wrote:

Also, I suppose that we could have an API too, if all that the API did was modify the underlying model.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/xonsh/xonsh/issues/1204#issuecomment-228642807, or mute the thread https://github.com/notifications/unsubscribe/AADlQH6rk_pESDxELe5vG4zOAfSWVI-uks5qPzgigaJpZM4IzUsQ .

[scopatz]

@PaulReiber - That seems like it is a separate topic, if you'd like to open one up. Also it seems like this is similar in idea to what plumbum does. And autoplumber is an interesting concept. Maybe that could be a xontrib or if we could figure out a way to automatically insert names into the context based on the current __xonsh_commands_cache__ keys... I am not sure this is 100% safe all of the time (would have to see an implementation) but I think it is certainly worth a discussion in another thread.

[scopatz]

In general, I like the idea of hooks, though other than needing a post-load one for the way things currently work, I can't think of an event I would need to hook onto

precmd and postcmd hooks are something that @asmeurer and discussed a while back. For example, this would allow iTerm2 users to hook into its metadata system. We could probably even implement setting the title as a pre-command hook.

The more I think about it, the more I like a $PATH / EnvPath kind of approach. Users can set and modify elements as if it was a mutable sequence. However, there would also be alternative views into this data structure and decorators that would register functions into this instance.

If we go down this path, I would be tempted to go deprecate $XONSHRC and $XONSHCONFIG in favor of just $XONSHOOKS or similar.

[scopatz]

Or we could just consolidate everything into $XONSHRC

[scopatz]

This has been implemented (and more!) via the evenys system. Please feel free to reopen if needed.