Which step would one take to integrate an existing Makefile with moulti?

[MartyLake]

Hello, Thanks for creating this software! Which step would one take to integrate an existing Makefile with moulti?

One possibility might be to use script scrip.txt make configure build to get the make output to go to a text fail, then tail that file to moulti. However, how to differentiate Makefile steps ? Should I instrument the makefile to emit markers ?

Also, how to do all that in the same terminal instead of multiple ones?

Best,

[xavierog]

Which step would one take to integrate an existing Makefile with moulti?

One possibility might be to use script script.txt make configure build to get the make output to go to a text file

Do you really need script for that? Regular output redirection should do, especially if you can convince your tools to keep outputting ANSI colors.

then tail that file to moulti.

Same: technically, you do not need an intermediate file (but feel free to generate one anyway if that makes sense in your workflow).

However, how to differentiate Makefile steps ? Should I instrument the makefile to emit markers ?

I see two main approaches: Approach 1: editing the Makefile so it runs moulti step add whenever necessary; but then it becomes necessary to pipe all output to a moulti pass command until the next moulti step add. I'll have a look at the GNU make documentation but right now, that does not strike me as a simple and straightforward approach.

Approach 2: as you suggest, having markers in the output of make makes it easier to write a wrapper. Actually, one of my ideas for the future of Moulti is to provide a wrapper tool that reads a format like that:

===== divider1 =====
Lorem ipsum dolor sit amet, consectetur adipiscing elit.
##### title1 #####
# moulti: classes=warning min_height=3 max_height=3 top_text="this format still" bottom_text="needs some work"
Curabitur ultricies dolor augue, id eleifend erat varius ut.
Etiam eget dapibus lectus, in gravida sapien. Aliquam nec diam mi.
Interdum et malesuada fames ac ante ipsum primis in faucibus.
##### title2 #####
# moulti: classes=success
Mauris in pellentesque lorem. Nam id libero at libero malesuada euismod non vel turpis.
Aenean dolor sem, suscipit at risus sed, tempus blandit nisl. Nullam ut tincidunt metus, non interdum urna. 

... and turns each title into a Moulti step. That way, Moulti users could "simply" reformat the output of their existing scripts to integrate them with Moulti. This is essentially a less specialized variant of what moulti diff does already.

Also, how to do all that in the same terminal instead of multiple ones?

Although the quickstart makes you play with two terminals at first, the documentation also details moulti run in the Shell scripting section.

[MartyLake]

Do you really need script for that? Regular output redirection should do, especially if you can convince your tools to keep outputting ANSI colors. Same: technically, you do not need an intermediate file (but feel free to generate one anyway if that makes sense in your workflow).

I would very much like to avoid doing so, I am happy to read that you think it can be avoided altogether.

I see two main approaches: Approach 1: editing the Makefile so it runs moulti step add whenever necessary; but then it becomes necessary to pipe all output to a moulti pass command until the next moulti step add. I'll have a look at the GNU make documentation but right now, that does not strike me as a simple and straightforward approach.

This is what I undertood I should do reading the Documentation.md, and it seemed unrealistic to create a whole new makefile system to accomodate for moulti.

Although the quickstart makes you play with two terminals at first, the documentation also details moulti run in the Shell scripting section.

See answer for Approach1.

Approach 2: as you suggest, having markers in the output of make makes it easier to write a wrapper. Actually, one of my ideas for the future of Moulti is to provide a wrapper tool that reads a format like that:

===== divider1 =====
Lorem ipsum dolor sit amet, consectetur adipiscing elit.
##### title1 #####
# moulti: classes=warning min_height=3 max_height=3 top_text="this format still" bottom_text="needs some work"
Curabitur ultricies dolor augue, id eleifend erat varius ut.
Etiam eget dapibus lectus, in gravida sapien. Aliquam nec diam mi.
Interdum et malesuada fames ac ante ipsum primis in faucibus.
##### title2 #####
# moulti: classes=success
Mauris in pellentesque lorem. Nam id libero at libero malesuada euismod non vel turpis.
Aenean dolor sem, suscipit at risus sed, tempus blandit nisl. Nullam ut tincidunt metus, non interdum urna. 

... and turns each title into a Moulti step. That way, Moulti users could "simply" reformat the output of their existing scripts to integrate them with Moulti. This is essentially a less specialized variant of what moulti diff does already.

That would be ideal! the moulti instructions could even be hidden behind a "erase current line" terminal command so that it is not visible when running on a system where moulti is not installed.

I had a great experience with the klogg app https://klogg.filimonov.dev/ where you can open any log, then search for things and matched lines would list in a window bellow. You could also define regexes and styles to be applied (fg / bg color, bold etc).

I guess one direction moulti development could take is follow that approach where a user create some configuration file that corresponds to the way its existing scripts already work, and then wrap the launch of such script.

For example, launching make with -d flags yields some interesting strings to grep:

    Must remake target `build-Debug'.
ninja: Entering directory `output/MyApp/Debug'

[0/20] SOLINK libMyAppLib.dylib, POSTBUILDS
[1/20] SOLINK libMyAppLib.dylib, POSTBUILDS
    Successfully remade target file `build-Debug'.
   Finished prerequisites of target file `build'.
  Must remake target `build'.
  Successfully remade target file `build'.
 Finished prerequisites of target file `all'.
Must remake target `all'.
Successfully remade target file `all'.

Here the steps are started by "Must remake target" and completed by "Successfully remade target", where "[X/Y]" is the progress of such target.

If moulti has some DSL to express that in the command line, or with some ad-hoc config file ,it would be easier to integrate with existing machinery.

Or I could create a filter that calls moulti appropriately when such strings appear, but I am not even sure how to do that.

[xavierog]

Or I could create a filter that calls moulti appropriately when such strings appear, but I am not even sure how to do that.

Like this?

moultimake.bash:

#!/usr/bin/env bash

# Instruct the Moulti instance to harvest any output not explicitly assigned to
# a step:
export MOULTI_RUN_OUTPUT=harvest
export MOULTI_INSTANCE="make"
[ "${MOULTI_RUN}" ] || exec moulti run -- "$0" "$@"

function process_lines {
    local step_counter=0
    while read -r line; do
        if [[ "${line}" =~ 'Must remake target' ]]; then
            # New target: create a new Moulti step:
            local title=$(sed 's/Must remake target `\(.*\)'\''./\1/' <<< "${line}")
            ((++step_counter))
            local step_id="make_$$_${step_counter}"
            moulti step add "${step_id}" --title="${title}" --bottom-text=' ' < /dev/null
            # Redirect stdout to `moulti pass our_new_step`
            exec > >(moulti pass "${step_id}")
        fi
        [[ "${line}" =~ 'Successfully remade target' ]] && moulti step update "${step_id}" --classes=success
        [[ "${line}" =~ error ]] && moulti step update "${step_id}" --classes=error
        printf '%s\n' "${line}"
    done
}

# Instance title:
moulti set --title="$*"

# Run the command given on the command-line and process each line of its
# output:
"$@" 2>&1 | process_lines

Usage:

chmod +x moultimake.bash
./moultimake.bash make -d

Regarding the rest of your suggestions: thank you, I will take all of this into account when designing the next features. The thing with Moulti is: there will always be more that one way to do it. Right now, Moulti requires writing a controlling script like the one above, except for a couple of uses (like Ansible playbooks and unified diff) where it already provides wrappers out of the box. As time goes on, Moulti should provide more of such wrappers, including some matching your use cases.

[MartyLake]

It works!!! That’s so cool! I think that example that you just created can easily be adapted to other "filter" use cases, and should be put in the documentation somewhere close to the beginning. :)

[xavierog]

It works!!! That’s so cool!

Glad you like it.

I think that example that you just created can easily be adapted to other "filter" use cases, and should be put in the documentation somewhere close to the beginning. :)

It will definitely serve as a basis for my next reflections. Specifically:

• I can turn it into some kind of mini-framework for users who prefer imperative approaches.
• it is a good tool to explore what a declarative approach should cover -- e.g. you can already notice a number of arbitrary decisions I took when writing this script: a good declarative format should provide enough flexibility to account for most, if not all use cases.

[MartyLake]

Glad you like it.

Yes! And it was easy now with this template to complete with some progress bar \o/

see grep and sed lines:

#!/usr/bin/env bash

# Instruct the Moulti instance to harvest any output not explicitly assigned to
# a step:
export MOULTI_RUN_OUTPUT=harvest
export MOULTI_INSTANCE="make"
[ "${MOULTI_RUN}" ] || exec moulti run -- "$0" "$@"

function process_lines {
    local step_counter=0
    while read -r line; do
        if [[ "${line}" =~ 'Must remake target' ]]; then
            # New target: create a new Moulti step:
            local title=$(sed 's/Must remake target `\(.*\)'\''./\1/' <<<"${line}")
            ((++step_counter))
            local step_id="make_$$_${step_counter}"
            moulti step add "${step_id}" --title="${title}" --bottom-text=' ' </dev/null
            # Redirect stdout to `moulti pass our_new_step`
            exec > >(moulti pass "${step_id}")
        fi
        if echo ${line} | grep -q '^\[[0-9]*\/[0-9]*\]'; then
            local progress_status=$(echo ${line} | sed -e 's/^\[\([0-9]*\)\/\([0-9]*\).*/\1/')
            local progress_target=$(echo ${line} | sed -e 's/^\[\([0-9]*\)\/\([0-9]*\).*/\2/')
            moulti set --progress-bar --progress="${progress_status}" --progress-target="${progress_target}"
        fi
        [[ "${line}" =~ 'Successfully remade target' ]] && moulti step update "${step_id}" --classes=success && moulti set --no-progress-bar
        [[ "${line}" =~ error ]] && moulti step update "${step_id}" --classes=error
        printf '%s\n' "${line}"
    done
}

# Instance title:
moulti set --title="$*"

# Run the command given on the command-line and process each line of its
# output:
"$@" 2>&1 | process_lines

Right now, when some steps have happened, I cannot know what happens or happened until I scroll with the mouse over moulti.

klogg has a follow mode where the window scrolls to the latest line. that mode is toggled by pressing f.

While moulti already scrolls inside steps, could moulti also have something similar to scroll finished/inactive steps ? (should I open new issues?)

It will definitely serve as a basis for my next reflections. Specifically:

* I can turn it into some kind of mini-framework for users who prefer _imperative_ approaches.

That would be my case I guess.

* it is a good tool to explore what a _declarative_ approach should cover -- e.g. you can already notice a number of arbitrary decisions I took when writing this script: a good declarative format should provide enough flexibility to account for most, if not all use cases.

When I see how concise this moultimake.bash is, I wonder if it is necessary at all. Do you have any idea how a declarative file would look like?

[xavierog]

Right now, when some steps have happened, I cannot know what happens or happened until I scroll with the mouse over moulti.

klogg has a follow mode where the window scrolls to the latest line. that mode is toggled by pressing f.

While moulti already scrolls inside steps, could moulti also have something similar to scroll finished/inactive steps ? (should I open new issues?)

Moulti has programmatic scrolling too. Refer to the documentation for details but TL;DR:

• add --scroll-on-activity=-1 to the moulti step add command
• hit 'l' to suspend programmatic scrolling (the main scrollbar turns green)
• hit 'l' again to restore it (the main scrollbar turns grey again)

[xavierog]

When I see how concise this moultimake.bash is, I wonder if it is necessary at all.

That will be part of my reflections too.

Do you have any idea how a declarative file would look like?

The default approach would be a serializable Python dict written as JSON/YAML/TOML that describes:

• named patterns (e.g. section_title: '^#{10} (?<title>.+) #{10}$')
• pattern groups (typically optional, purely for the sake of convenience)
• actions (e.g. new step with these parameters, new divider with those options, set instance title or progress bar status)
• action groups (typically optional, purely for the sake of convenience)
• pattern/action associations (e.g. "upon reading a pattern from this pattern group, run those action groups")
• what time to trigger the coffee machine

+ support for using regex captures and environment variables in actions.

I could implement that by myself OR I could look for a tool/library/framework that already provides this kind of logic.

[MartyLake]

Moulti has programmatic scrolling too. Refer to the documentation for details but TL;DR:

• add --scroll-on-activity=-1 to the moulti step add command

That’s very neat! I wonder how I missed that when searching in the documentation. Thanks!

* hit 'l' to suspend programmatic scrolling (the main scrollbar turns green)
* hit 'l' again to restore it (the main scrollbar turns grey again)

I tried those, but I have troubles understanding if scroll stops, or if scripts takes time. I will use --scroll-on-activity for now anyway so I will figure this later.

The default approach would be a serializable Python dict written as JSON/YAML/TOML that describes:

* named patterns (e.g. `section_title: '^#{10} (?<title>.+) #{10}$'`)
* pattern groups (typically optional, purely for the sake of convenience)
* actions (e.g. new step with these parameters, new divider with those options, set instance title or progress bar status)
* action groups (typically optional, purely for the sake of convenience)
* pattern/action associations (e.g. "upon reading a pattern from this pattern group, run those action groups")
* what time to trigger the coffee machine

• support for using regex captures and environment variables in actions.

I could implement that by myself OR I could look for a tool/library/framework that already provides this kind of logic.

Reading all of this, I wonder if a python API might be easier to implement (and more powerful) than JSON/YAML/TMOL.

This api would read like declarative, but really just call python functions, a bit like the cmake syntax.

import moulti as m

m.detector.add_new_step_rule("Must remake target `\(.*\)'\'", auto_increment_id=True, show_progress_bar=False)
m.detector.add_step_success_rule(".*Successfully remade target.*")
m.detector.add_step_error_rule(".*error.*")
m.detector.add_step_status_progress_rule("^\[\([0-9]*\)\/[0-9]*.*", force_show_progress_bar=True)
m.detector.add_step_total_progress_rule("^\[[0-9]*\/\([0-9]*\).*", force_show_progress_bar=True)

m.add_action(["make", "-d", "configure"], can_be_restarted=True, on_success="next_command")
m.add_action(["make", "-d", "build"], can_be_restarted=True, on_success="next_command")
m.add_action(["make", "-d", "test"], can_be_restarted=True)

m.run(run_output="harvest", instance="make", start_instance_if_needed=True, follow=True)

Later part can easily be made customizable using vanilla python:

for i in range(1,len(argv)):
  m.add_restartable_command(["make", "-d", argv[i)], can_be_restarted=True, on_success="next_command")
m.run(run_output="harvest", instance="make", start_instance_if_needed=True, follow=True)

[xavierog]

I tried those, but I have troubles understanding if scroll stops, or if scripts takes time. I will use --scroll-on-activity for now anyway so I will figure this later.

They are complementary features: --scroll-on-activity provides the automatic scrolling you expect. But sometimes, you may want to read a part of the output while your job is still running. But how can you read step number 5 while Moulti insists on scrolling to display step number 78? The "lock scroll" action enables you to scroll by yourself without being disturbed by the programmatic scrolling.

[xavierog]

Reading all of this, I wonder if a python API might be easier to implement (and more powerful) than JSON/YAML/TMOL.

It is planned to implement a "Python User API", i.e. a Python API exposed to users and that would adhere to semver. It will definitely be more powerful (e.g. it should integrate a class that keeps track of all created steps, and it should not need as many fork-exec-connect-send-receive-close-exit operations as a shell script) but this will likely wait until things get more stable (because a stable feature set makes it easier to design a perennial API).

My reflections so far:

• re: the idea of an output format that Moulti would recognize out of the box: it still makes sense but ideally, this idea should be implemented using one of the ideas below, possibly serving as example/demo.
• re: the idea of a framework for imperative-style parsing/translation: one issue is that there are too many languages and I am reluctant to introduce a new Domain-Specific Language. Considering how it quickly and concisely solved your own issue, I am tempted to make bash a priority: adding a few functions to moulti-functions.bash, demonstrating their use in the documentation and ending with a solution that is even more concise.
• re: the idea of a declarative-style tool: the idea remains relevant precisely because it is independent of any programming language. It still needs exploration, but I already pinpointed a use case that requires a Finite-State Machine and thus brings complexity: parsing such titles:

  ==========================================================
  This is a title only because it is surrounded with = signs
  ==========================================================
  This line should go inside a Moulti step.
  This line should go inside a Moulti step.
  This line should go inside a Moulti step.
  ================================================================
  This is another title only because it is surrounded with = signs
  ================================================================
  This line should go inside a Moulti step.
  This line should go inside a Moulti step.
  This line should go inside a Moulti step.

By the way, real life use cases are most welcome to feed those reflections. Do you have any other use case in mind?