The commands can't be executed back to back

[irbian]

Describe the bug I'm trying to automatize the services management and I´m having problems executing multiple pelias commands on my bash script

I have something like

pelias import wof pelias import oa pelias import osm

But it doesn´t work. It launch pelias import wof but pelias import oa and osm looks like they just don´t execute

I have had luck with something like

pelias import wof && pelias import oa && pelias import osm

Environment (please complete the following information): Linux

[missinglink]

Bash scripts run serially by default.

The && syntax is equivalent, it simply applies a boolean AND condition.

So the two methods above are equivalent and should produce the same result.

The pelias command is just a thin wrapper around the docker-compose and docker commands.

You can execute docker ps -a at any point to list the running containers.

I don't believe this is a bug as many other developers have been using these commands as-is for several years without issue.

[edit] spelling

[irbian]

I understand your point. I did some tests and found that

If I create a test.sh where the contents are

pelias import oa
pelias import osm
pelias import wof

, and I execute the script, it works perfectly. BUT and this si what I have been using, if I copy/paste the commands so they execute in order one after the other OR if I use them inside a heredoc send via an SSH, it breaks. It looks like it takes the second and third command as an input for the first, so they don´t execute

pelias@mymachine:/mypelias$ pelias import wof
pelias import oa
pelias import osminfo: [whosonfirst] import finished
info: [dbclient-whosonfirst] paused=false, transient=3, current_length=0
info: [dbclient-whosonfirst] paused=false, transient=3, current_length=0
info: [dbclient-whosonfirst] paused=false, transient=0, current_length=0, indexed=1051, batch_ok=3, batch_retries=0, failed_records=0, locality=959, country=1, macroregion=1, region=2, localadmin=88, persec=105.1
info: [dbclient-whosonfirst] paused=false, transient=0, current_length=0, indexed=1051, batch_ok=3, batch_retries=0, failed_records=0, locality=959, country=1, macroregion=1, region=2, localadmin=88, persec=105.1

This don´t happen with something similar like

docker run --rm hello-world
docker run --rm hello-world
docker run --rm hello-world

At this point I understand that is something more about my own ignorance with linux and not something specific of pelias, but as you say the pelias command is not more than a wrapper so I don´t know where the difference comes from

[irbian]

Retaking this issue, is maybe related to the wait you have here? https://github.com/pelias/docker/blob/master/cmd/prepare.sh#L20

In that, when we call prepare_all it would work perfectly because the wait is included, but when the prepare_placeholder is called on it's own it don´t work because it lacks the wait command?

[missinglink]

The linux wait command only applies when tasks have been sent to the background using the & symbol. In this case the wait command will block script execution until the commands in the background have completed.

[missinglink]

and I execute the script, it works perfectly. BUT and this si what I have been using, if I copy/paste the commands so they execute in order one after the other OR if I use them inside a heredoc send via an SSH, it breaks. It looks like it takes the second and third command as an input for the first, so they don´t execute

Can you please explain a bit more detail about this comment, particularly with an example of the heredoc syntax you're using.

From what you've said "It looks like it takes the second and third command as an input for the first" it sounds like you're missing the newline between commands.

[irbian]

and I execute the script, it works perfectly. BUT and this si what I have been using, if I copy/paste the commands so they execute in order one after the other OR if I use them inside a heredoc send via an SSH, it breaks. It looks like it takes the second and third command as an input for the first, so they don´t execute

Can you please explain a bit more detail about this comment, particularly with an example of the heredoc syntax you're using.

From what you've said "It looks like it takes the second and third command as an input for the first" it sounds like you're missing the newline between commands.

🤔 maybe in some place on the flow the new lines are deleted so it executes as a big single line? and maybe the affected commands are the ones that allow for params to be received so it just makes that? and my && force them to be interpreted as different commands? mmm maybe, I´ll let you know if I figure it out

[missinglink]

Here's a visual explanation.

By default the commands run serially such as this:

function sync(){       
  prepare_polylines    
  prepare_placeholder  
}                      

          │            
          ▼            
   ┌─────────────┐     
   │  polylines  │     
   └─────────────┘     
          │                        
          ▼            
  ┌───────────────┐    
  │  placeholder  │    
  └───────────────┘    
          │                       
          ▼            

if your commands end with either a newline \n, a semicolon ; or 'logical and' && the parser stops trying to read any further argument or flags from the script and will run the command proceeding the symbol immediately in serial mode.

The other option is to run the commands in parallel by backgrounding the tasks and then using wait to synchronise them again, this is not the default mode and is only applicable when the 'backgrounding operator' & is used.

function async(){            
  prepare_polylines &        
  prepare_placeholder &      
  wait                       
}                            

                │                  
       ┌───┬────┴─────┬───┐        
       │ & │          │ & │        
       ├───┘          └───┤        
       │                  │        
       ▼                  ▼        
┌─────────────┐   ┌───────────────┐
│  polylines  │   │  placeholder  │
└─────────────┘   └───────────────┘
       │                  │        
       └───────┐┌─────────┘        
               ││                  
               ▼▼                  
          ┌──────────┐             
          │   wait   │             
          └─────┬────┘             
                │                                   
                ▼                  

[missinglink]

If you're having success with &&, I'd suggest you replace it with ; which indicates an end-of-command even if the newline is absent. AFAIK a semi-colon is equivalent to a newline, so if it works with a semi-colon but not a newline then it's likely something is stripping the newlines.

The && isn't 'bad' per-se, it just has additional logic applied where the second command only runs if the first succeeds.

[missinglink]

Worth mentioning that there are several modes for heredocs, you might have some luck by modifying the syntax you're using.

A quick google shows some other developers talking about heredocs stripping newlines 🤷 https://unix.stackexchange.com/questions/534055/preserving-backslash-newlines-in-heredocs-run-in-a-subshell

[irbian]

You have been really helpful

That last question I too arrived. I have tried a different approach where I paste all the pelias commands on a script that I upload and execute and that looks like it make it more robust

My && approach didn´t always work. As you say, I think it was failing because one of them failed so the others didn´t execute, so it hanged before showing the error? I think I had to many parts and I´m missing some interactions, the main culprit I guess would be the ssh & heredocs

Anyway, thanks for your comments, I´m moving forward with the "bash setupPelias.sh" approach

[missinglink]

Ok great :+1:

If there's no bug in Pelias please close this issue.