make with_ loops configurable

[bcoca]

ISSUE TYPE

Feature Idea

COMPONENT NAME

core

ANSIBLE VERSION

2.1

CONFIGURATION

OS / ENVIRONMENT

SUMMARY

how: 
    forks: 1
    pause: 0
    squash: name
    label: "{{item.name}}"
    end: on_fail
with_items: ...

• forks: forks within the loop to do items in parallel, default 1, this needs warnings
• pause: between loop executions, useful in throttled api scenario Done in 2.2
• squash: join all items into list and pass to provided option, works like current hardcoded opts for apt, yum, etc, by default it should be None abandon: reversed opinion, we should remove this feature
• end: when to interrupt the loop, default is 'last item', options? on_fail, on_success (first one)?
• label: (#13710) what to display when outputting the item loop Done in 2.2

docs to current state at:

http://docs.ansible.com/ansible/playbooks_loops.html#loop-control

STEPS TO REPRODUCE

EXPECTED RESULTS

ACTUAL RESULTS

[amenonsen]

Please let's not call it how. That's even worse to read than become: true. But the functionality under it looks great.

[bcoca]

includes fix for #10695

[mahemoff]

Excellent. In the interests of bikeshedding, maybe call it looping:.

[realcnbs]

:+1:

[yikaus]

+1

[jyennaco]

+1 especially for within host parallelization!

[rbarabas]

:+1:

[hloeffler]

:+1: but lets not call it "how"

[bcoca]

so here is a workaround for breaking a loop task after the first failure

- hosts: localhost
  vars:
    myvar:
        - 1
        - 2
        - 3
        - 4
        - 5
  tasks:
    - name: break loop after 3
      debug: msg={{item}}
      failed_when: item == 3
      register: myresults
      when: not (myresults|default({}))|failed
      with_items: "{{myvar}}"

[yikaus]

@bcoca not working from end (ansible 1.9.3 ubuntu )

TASK: [break loop after 3] **** ok: [localhost] => (item=1) => { "failed": false, "failed_when_result": false, "item": 1, "msg": "1" } ok: [localhost] => (item=2) => { "failed": false, "failed_when_result": false, "item": 2, "msg": "2" } failed: [localhost] => (item=3) => {"failed": true, "failed_when_result": true, "item": 3, "verbose_always": true} msg: 3 ok: [localhost] => (item=4) => { "failed": false, "failed_when_result": false, "item": 4, "msg": "4" } ok: [localhost] => (item=5) => { "failed": false, "failed_when_result": false, "item": 5, "msg": "5" }

[bcoca]

ah, yes, it will work as is in 2.0 as in 1.9 the registration does not occur until after the loop is done.

[mattyb]

+1 on forks

[t1m0thy]

+1 perhaps instead of "how", loop_details or options?

[daniel-sc]

+1

[senderista]

+1, using wait_for from localhost to wait for 100 EC2 hosts to come up is killing me...

[beholt]

+1 for similar reason to senderista

[ykuksenko]

+1

[evenme]

:+1:

[kustodian]

Both squash and forks would be awesome features which would speed up Ansible execution immensely.

I would also replace how with something like loop_details, loop_settings, loop_options, or anything similar.

[bcoca]

loop_control , already in 2.1 with the label part implemented.

squash might just go away as it is easy to just pass a list to the modules that support it:

apt: name={{listofpackages}}

and avoid the loop completely

[mahemoff]

http://docs.ansible.com/ansible/playbooks_loops.html#loop-control

[jeanpaulmanjarres-payulatam]

+1 forks

[ansafonov]

+1 forks

[resmo]

I had a use case for a new config for a conditional break break_when

[kasabov]

+1 forks and I hope it'll also work for parallelizing sequences of tasks to run as in: - include: service.yml with_items: "{{services|default([])}}"

Otherwise, there's the async/async_status already.

[bryfry]

+1 forks

Me waiting for openstack modules to iterate through with_ loops on 100+ items...

[ghjm]

+1 on forks. Could use this for copying AMIs to all the AWS regions.

[aarlint]

+1 on forks. Please! make it part of loop_control

[uebertrieben]

+1 on forks

[ghost]

+1 on forks

[Schroeffu]

+1, need forks too :-)

[Haxe18]

+1 on forks

[pschorf]

+1 on forks

[berendt]

Forks would be awesome, +1

[xenithorb]

I always sit in silence not wanting to contribute to the spam, because it's hard to gauge between projects whether it's welcomed or not, but it looks like the fork +1 train has left the station!

+1 on forks

[adrien-mogenet]

+1 on forks

[ImpSy]

+1 on forks

[hai-ld]

:+1: on forks

[ansibot]

@bcoca Greetings! Thanks for taking the time to open this issue. In order for the community to handle your issue effectively, we need a bit more information.

Here are the items we could not find in your description:

• issue type
• ansible version
• component name

Please set the description of this issue with this template: https://raw.githubusercontent.com/ansible/ansible/devel/.github/ISSUE_TEMPLATE.md

click here for bot help

[d3matt]

+1 on forks

[inponomarev]

+1 on forks!

[ahharu]

+1 on Forks!

[bluefish6]

+1 on forks!

[jt1]

+1 on forks!

[bitliner]

Any update on fork? When is it supposed to be ready? My use case is instantiating about 20 containers on a server with with_sequence. Now it takes too many ages :( I would be glad to help, but I would need some tips on where to put hands

[bcoca]

@bitliner no one has created a PR for it, if that is what you are asking, its actually very hard to do correctly.

as for your issue, just declare X hosts in inventory and loop over hosts: instead of with_ to create them in paralell.

inventory:

[containers]
container[000:020]

hosts: containers
gather_facts: false
tasks:
   - container: state=present name={{inventory_hostname}}

i.e container is a 'made up' module.

[bitliner]

@bcoca your solution is not clear to me. To be sure, is this what you mean?

hosts file

[containers]
192.168.1.100
192.168.1.100
192.168.1.100
192.168.1.100
... and so on based on the degree of parallelism I want to get ...

main.yml file

hosts: containers
gather_facts: false
tasks:
   - container: state=present name={{inventory_hostname}}

based on a container module that I should implement, correct? In this case, I would have all containers with the same name, and that is not acceptable, correct?

Furthermore, what are the challenges to do implement loop in parallel correctly?

My use case needs to speed up this task:

- name: "Start clients"
  docker_container:
    name: "mycontainer-{{ item }}"
    image: myimage
    links: 
      - server-{{item}}:server-{{item}}
  with_sequence: count={{ scale }}

I can't use docker-compose scale because I need to route traffic among containers in a specific way (that is why I use with_sequence to generate different docker container names).

I could build a module that takes the declaration of a container and based on a scale parameter it instantiates remotely multiple containers in parallel. Does it make sense? Do you have any help to understand how to call/re-use in myModule the docker_container module and what are the API that Ansible offers to run something in parallel?

[bcoca]

@bitliner you did not follow my instructions, i had unique names in inventory (using range to work just like a sequence). As names are unique in inventory, you just declared the same name N times, but still have one host, which does not cause your 2nd issue on dupe names of containers as you only loop over 1 host.

In any case, if you want to follow up with your issue, use ML or IRC as it would be a bit off topic for this ticket.

[bitliner]

@bcoca How can I have one host if you declared 20 hosts?

[containers]
container[000:020]

is going to connect to container001, container002, etc.

It works for having unique names, but what is not clear to me is why you say I still have one host (instead of 20). hosts:containers means to me 20 hosts, not just one. What am I ignoring in this solution?

[bcoca]

@bitliner cause this:

[containers]
192.168.1.100
192.168.1.100
192.168.1.100
192.168.1.100

is not 4 hosts, but 1

also at this point this is pretty much off topic, if you want to continue getting help on this go to IRC or ML