nushell / engine-q

MIT License
58 stars 40 forks source link

Porting commands to engine-q list #242

Closed stormasm closed 2 years ago

stormasm commented 3 years ago

Updated on January 14, 2022

We are getting closer thanks to everyone's help !

Instead of commenting on this issue, please go to nushell/nushell#4356

Engine-q will become Nushell's new parsing and evaluation engine in the future. As part of this, we'll port the existing Nushell commands into Engine-q. Much of this porting can be done mechanically, as while the two systems differ, they are similar enough that many commands can be ported following the steps below.

You can sign up to help out porting a command by commenting on this issue.

Our preference when sending us a Pull Request (PR) is to do one command at a time. It makes it a lot easier for us to review the code as well as making the merge conflicts easier to figure out...

Note: Instructions on how to port these commands from Nushell to Engine-q are below the list.

name sub category completed notes
alias core done
all? done @arthur-targaryen
ansi done @matthewauld
ansi gradient done @matthewauld
ansi strip done @matthewauld
any? done @arthur-targaryen
append filters done @stormasm
autoenv @jntrnr
autoenv trust @jntrnr
autoenv untrust @jntrnr
autoview done @jntrnr - mostly built into table now
benchmark system done
build-string strings done
cal done @onthebridgetonowhere
cd filesystem almost done missing cd -
char done @matthewauld
chart
clear done @Scorpil
collect done
compact done @matthewauld
cp filesystem done
dataframe done @elferherrera
date done @onthebridgetonowhere
date format done @onthebridgetonowhere
date humanize done @onthebridgetonowhere
date list-timezone done @onthebridgetonowhere
date now done @onthebridgetonowhere
date to-table done @onthebridgetonowhere
date to-timezone done @onthebridgetonowhere
debug done
def core done
default @matthewauld
describe done
do core done
drop filters done @stormasm
drop column filters done @stormasm
drop nth filters done @stormasm
du @Scorpil
each filters done
each group filters
each window filters
echo core done @aslynatilla
empty? done @onthebridgetonowhere
enter done @jntrnr
every done @Pantonshire
exec @Scorpil
exit done @jntrnr
fetch done @onthebridgetonowhere
find @arthur-targaryen
first filters done @onthebridgetonowhere. Basic support in. Binary not yet supported
flatten done @onthebridgetonowhere
for core done
format strings done @aslynatilla
format filesize
from formats done
from csv formats done @jntrnr
from eml formats done @luccasmmg
from ics formats done @luccasmmg
from ini formats done @luccasmmg
from json formats done
from ods formats done @luccasmmg
from ssv formats done @luccasmmg
from toml formats done @luccasmmg
from tsv formats done @jntrnr
from url formats done @luccasmmg
from vcf formats done @luccasmmg
from xlsx formats done @luccasmmg
from xml formats done @luccasmmg
from yaml formats done @luccasmmg
from yml formats done @luccasmmg
g done
get filters done
group-by @andrasio
group-by date
hash done @CBenoit
hash base64 @CBenoit
hash md5 done @CBenoit
hash sha256 done @CBenoit
headers
help core mostly done Still needs syntax highlighting
histogram
history
if core done
ignore done @hustcer
insert
into conversions done
into bool conversions done
into binary conversions done
into column_path conversions
into datetime conversions done
into decimal conversions done
into filesize conversions done
into int conversions done
into path conversions done Done? No longer necessary
into string conversions done @onthebridgetonowhere
keep until done @arthur-targaryen
keep while done @arthur-targaryen
kill done @Scorpil
last filters done @stormasm
length filters done
let core done
let-env done
lines filters done
load-env
ls filesystem done @jntrnr
math done @luccasmmg is working on all of the math commands below
math abs done @luccasmmg
math avg done @luccasmmg
math ceil done
math eval done
math floor done
math max done @luccasmmg
math median done
math min done @luccasmmg
math mode done
math product done
math round done
math sqrt done
math stddev done
math sum done
math variance done @luccasmmg is working on all of the math commands above here
merge
mkdir filesystem done
move
mv filesystem done
n done @jntrnr
nth filters done @stormasm
open done @jntrnr
p done @jntrnr
parse done @CBenoit
path done @LiHRaM
path basename done @LiHRaM
path dirname done @LiHRaM
path exists done @LiHRaM
path expand done @LiHRaM
path join done @LiHRaM
path parse done @LiHRaM
path relative done @LiHRaM
path split done @LiHRaM
path type done @LiHRaM
pathvar done @kubouch switched e-q to using Value instead of String for env vars
pivot
post
prepend filters done @stormasm
ps done
pwd @Scorpil
random done @jaeheonji
random bool done @jaeheonji
random chars done @jaeheonji
random decimal done @jaeheonji
random dice done @jaeheonji
random integer done @jaeheonji
random uuid done @jaeheonji
range filters done @stormasm
reduce @kubouch
reject filters done @stormasm
rename filters
reverse filters done @stormasm
rm done
roll
roll column
roll up
rotate
rotate counter-clockwise
save
select filters mostly done Still needs to have latest changes ported from nushell
seq @nibon7
seq date done
shells done @jntrnr
shuffle filters done @stormasm
size done
skip done @arthur-targaryen
skip until done @arthur-targaryen
skip while done @arthur-targaryen
sleep done @Scorpil
sort-by filters @stormasm
source mostly done currently a bit buggy, needs some testing
split done
split chars done
split column done
split row done
split-by @andrasio
str done @onthebridgetonowhere
str camel-case done @onthebridgetonowhere
str capitalize done @onthebridgetonowhere
str collect done @onthebridgetonowhere
str contains done @onthebridgetonowhere
str downcase done @onthebridgetonowhere
str ends-with done @onthebridgetonowhere
str find-replace done @onthebridgetonowhere
str index-of done @onthebridgetonowhere
str kebab-case done @onthebridgetonowhere
str length done @onthebridgetonowhere
str lpad done @onthebridgetonowhere
str pascal-case done @onthebridgetonowhere
str reverse done @onthebridgetonowhere
str rpad done @onthebridgetonowhere
str screaming-snake-case done @onthebridgetonowhere
str snake-case done @onthebridgetonowhere
str starts-with done @onthebridgetonowhere
str substring done @onthebridgetonowhere
str trim done @onthebridgetonowhere
str upcase done @onthebridgetonowhere
sys system done
table viewers done output and theming now available
tags done Ported as metadata
to done
to csv done @luccasmmg
to html done @luccasmmg
to json done @jntrnr
to md done @luccasmmg
to toml done @luccasmmg
to tsv done @luccasmmg
to url done @luccasmmg
to xml done @luccasmmg
to yaml done @luccasmmg
touch filesystem done
tutor
unalias
uniq done @onthebridgetonowhere
unlet-env done now part of hide
update done
update cells
url done @jaeheonji
url host done @jaeheonji
url path done @jaeheonji
url query done @jaeheonji
url scheme done @jaeheonji
version done @onthebridgetonowhere
where filters done
which @Scorpil
with-env done @jntrnr
wrap filters done
zip done
$nu mostly done @jntrnr
$scope done this is done as a simple implementation, but feel free to improve it

Missing features/commands

Porting WholeStreamCommand (nushell) to Command (engine-q)

All commands have the standard boilerplate methods including

Name and usage is fairly obvious

Signature

The signature can have required parameters, optional parameters, or no parameters. An example of a command with no parameters is the command length.

  fn signature(&self) -> nu_protocol::Signature {
      Signature::build("length").category(Category::Filters)
 }

Please be sure and add in the Category of the command you are porting so that our documentation and filtering system picks it up

Commands with optional and required parts should work similarly between Nushell and Engine-q.

Run

Nushell:

fn run(&self, args: CommandArgs) -> Result<OutputStream, ShellError> { ... }

Engine-q:

    fn run(
        &self,
        engine_state: &EngineState,
        stack: &mut Stack,
        call: &Call,
        input: PipelineData,
    ) -> Result<nu_protocol::PipelineData, nu_protocol::ShellError> {

Run is where all of the action happens when porting over a command from nushell or writing a new command. You'll notice that Engine-q uses a much more expanded form of the arguments when compared to Nushell's more compact form. This allows for clear access to the components of running a command:

EngineState

The EngineState gives access to the global state of the engine, which currently is mostly the declarations that need to be known in order to run any code in the system.

Stack

The stack is a snapshot of the values that the current block can see. This Stack is narrowed using closure capture to only the variables the block can see.

Call

pub struct Call {
    /// identifier of the declaration to call
    pub decl_id: DeclId,
    pub head: Span,
    pub positional: Vec<Expression>,
    pub named: Vec<(String, Option<Expression>)>,
}

The call represents the call itself, including:

Rather than reaching into this structure directly, it's often better to access the contents of this call via the helper methods that can evaluate the unevaluated arguments for you:

// Example of getting a single required positional argument as a string with an associated span
let separator: Spanned<String> = call.req(context, 0)?;

The type of the variable allows call.req(...)? to invoke the right converter for you, failing with a type mismatch error if the caller has passed the wrong type at runtime. We use the same technique in Nushell, so this should port similarly with the change of having context as a new argument to the helper methods.

PipelineData

In Nushell, input and output were handled as streams. Commands often just applied themselves as calls onto these streams. The stream concept was helpful to get going, but also shows its limitations. For example, how do you represent a single value in the stream? What's the difference between a single item list and a single item? When should we take a list and flatten it into the stream? Ambiguities like this were fairly common in practice with implementing Nushell commands.

With Engine-q, we separate the various data types into specific cases. Single value outputs are now just a single value case of Value. Streams are handled separately from Value, and both form a PipelineData enum. This works for both input and output to commands.

To port from Nushell to Engine-q, for commands that call .map or .flat_map on the input stream, we've implemented helpers on Value that you can call with the same name. You can also call into_iter to get a full iterator. Additionalyl, you can also match on PipelineData yourself and handle the different cases.

Adding your commands

Add your newly ported commands to the crates/nu-command/src/default_context.rs file that is in the nu-command subcrate. There, you'll see a list of the commands that will be loaded in the default context. Adding your command here will make it visible when you start up Nu.

Tests and subcommands

If you add a new top-level command with subcommands, you'll need to add the top-level command to the group of default commands in the test context. You'll find this list in the crates/nu-command/src/example_test.rs file.

References

Core Engine-q data structures

From 12/13/2021 Discord

JT Note on Most important commands still needed to be ported

luccasmmg commented 3 years ago

I can do all the math commands no problem

stormasm commented 3 years ago

@luccasmmg sounds great ! thank you.... over time as you submit the PRs please reference this issue and then I will go ahead and mark it done ---- I will sign you up for the math commands now ---- so folks will know you are working on them...

vitvakatu commented 3 years ago

Hi, I want to try all of config commands

stormasm commented 3 years ago

@vitvakatu Thank you ! I signed you up for all of the config commands...

onthebridgetonowhere commented 3 years ago

I'd love to work on all the date commands!

sophiajt commented 3 years ago

@vitvakatu - engine-q config works a bit differently than Nushell, but you might be able to still port much of the functionality. Rather than using a toml file, engine-q uses a nushell script that runs in startup.

@onthebridgetonowhere - sounds good!

onthebridgetonowhere commented 3 years ago

Could I have a go at first as well? Thanks!

sophiajt commented 3 years ago

@onthebridgetonowhere - it's yours

aslynatilla commented 3 years ago

May I try and port echo and format?

sophiajt commented 3 years ago

@aslynatilla - they're yours. echo should be simpler in e-q than it was in Nushell as each does all the iteration now. You should just be able to send on the PipelineData you get

vitvakatu commented 3 years ago

@jntrnr yep, I've noticed this after I decided to work on config :) I think I will be able to port it anyway. Thank you!

onthebridgetonowhere commented 3 years ago

Could I also have into string from conversions? Thanks!

LiHRaM commented 3 years ago

Hi, please sign me up for the path commands, if possible. :)

sophiajt commented 3 years ago

@onthebridgetonowhere - yup! it's yours

@LiHRaM - definitely. they're yours

onthebridgetonowhere commented 3 years ago

Working on a few str cases:

luccasmmg commented 3 years ago

Can i be assigned the from yaml command?

sophiajt commented 3 years ago

@luccasmmg - yup, it's yours!

onthebridgetonowhere commented 2 years ago

Worked on the str capitalize command nushell/engine-q#317

onthebridgetonowhere commented 2 years ago

@jntrnr - I am planning to do all the str commands, but it will take me like a week or so (a bit every day). Unless, someone else wants to work on them and is faster :-) We can also mark into string as done, you added the cell path support to it.

sophiajt commented 2 years ago

@onthebridgetonowhere - cool I'll mark those.

kubouch commented 2 years ago

@onthebridgetonowhere Could you implement the str to-datetime and str to-decimal as into datetime and into decimal? I think they should be there and not in the str group.

onthebridgetonowhere commented 2 years ago

@kubouch let's do that. And would it keep its original purpose of converting strings only into datetime/decimal?

kubouch commented 2 years ago

@onthebridgetonowhere I think so. We have into X commands for other types as well. We might want to modify into decimal to accept also integers, not only strings, but that should be easy to do.

luccasmmg commented 2 years ago

Can i be assigned from url and from eml? also the math variance command it not marked in the issue, even tough it is done

sophiajt commented 2 years ago

@luccasmmg yup they're yours

CBenoit commented 2 years ago

Hello, I'm interesting in porting the parse command.

sophiajt commented 2 years ago

@CBenoit - it's yours

fdncred commented 2 years ago

@kubouch @onthebridgetonowhere related to the into commands, there is into int and str to-int and they have similar but different functionality. I'd like to see all of str to-int's functionality brought into into int if possible. Thoughts?

kubouch commented 2 years ago

@fdncred +1 definitely. I overlooked str to-int.

onthebridgetonowhere commented 2 years ago

@fdncred @kubouch we can do that! Thanks for the suggestion.

sophiajt commented 2 years ago

@vitvakatu - I ended up taking a look at the config from the engine side and put some basic support together for a config variable: https://github.com/nushell/engine-q/pull/332

I haven't merged it yet if you want to take a look.

I haven't ported any of the config commands, yet. Happy to chat about designs and how this works with your work.

vitvakatu commented 2 years ago

@jntrnr thank you for ping, I'll contact you in a few days if face some real problems. After a brief look, I think it won't notably affect my code

CBenoit commented 2 years ago

parse (category strings) command done porting in https://github.com/nushell/engine-q/pull/338 Next I’ll port hash command + subcommands.

luccasmmg commented 2 years ago

Can I be assigned the rest of the "from" commands?

sophiajt commented 2 years ago

@luccasmmg - yup!

jaeheonji commented 2 years ago

Could I contribute to random commands? 😸

stormasm commented 2 years ago

@jaeheonji great ! you are all signed up for the random commands... as you finish the commands reference this issue and we will mark them as done... We appreciate you helping us out, many thanks....

ghost commented 2 years ago

Hello, can I please be assigned to

stormasm commented 2 years ago

@arthur-targaryen you are all set and signed up for your requested commands... Thank you so much for helping out with our engine-q porting of nushell commands !

vitvakatu commented 2 years ago

@stormasm hi, I'm afraid I can't find time to port config commands in the nearest future, so please make these commands available for others. It's really cool to see so much activity on this task!

stormasm commented 2 years ago

@vitvakatu thanks for the heads up ! I removed your name from the config commands... Of course we welcome your participation in the future when you get more free time...

stormasm commented 2 years ago

@jaeheonji @arthur-targaryen

FYI thanks for your effort --- it will be easier if you do this...

Our preference when sending us a Pull Request (PR) is to do one command at a time. It makes it a lot easier for us to review the code as well as making the merge conflicts easier to figure out...

Thanks kindly !

ghost commented 2 years ago

Our preference when sending us a Pull Request (PR) is to do one command at a time.

Ok, I understand! I will split my PR into smaller ones.

luccasmmg commented 2 years ago

Since i'm almost finished with the from(only two left) commands, can i be assigned all the to as well?

sophiajt commented 2 years ago

@luccasmmg - yup!

Scorpil commented 2 years ago

Hi, I can take /platform, i.e. clear, du, exec, kill, pwd, sleep and which.

sophiajt commented 2 years ago

@Scorpil sounds good, I'll add you

onthebridgetonowhere commented 2 years ago

Phew, finished all the str commands. I can do cal and pathvar next.

kubouch commented 2 years ago

@onthebridgetonowhere Great job, thanks a lot! Maybe you can skip the pathvar, I had some ideas how we could get away without it.

onthebridgetonowhere commented 2 years ago

Sounds good! Will port cal then, I remember I worked on it in the original nushell.