Task parallelism on k8s

[eskuai]

Enhancement description: As a user, I want to launch the same task, running n times of instances/pods. All of them run independently.
I think that it is the k8s model parallel Jobs with a work queue: k8s Parallel Jobs

Solution description: It could be a new screen, "deploy task defininition" including values for generic deployer, as memory, cpu and --count-- and i check select job name with "count" value

Now, we can do task on k8s with meaning "Parallel Jobs with a work queue"

Description of alternatives:

Deployer on k8s should be upgrade to generate pod applying parallelism and concurrencypolicy.

apiVersion: batch/v1beta1
kind: CronJob
metadata:
  name: testcronjob
spec:
  schedule: "*/1 * * * *"
  jobTemplate:
    spec:
      template:
        spec:
          containers:
          - name: mycron-container
            image: alpine
            imagePullPolicy: IfNotPresent          
            command: ['sh', '-c', 'echo Job Pod is Running ; sleep 5']    
          restartPolicy: OnFailure
          terminationGracePeriodSeconds: 0
      parallelism: 3  
  concurrencyPolicy: Allow 

Additional context: Add any other context or explanation about the feature request here.

[mminella]

@eskuai , Can you provide a real world use case for what you are trying to accomplish?

[eskuai]

Hello @mminella

We need to get a lot of info from different sources (for examples, google ads, facebook, adobe analitys, etc...) and apply use cases as generating reports, writing data into hdfs for machine learning, analitycs, and more ...

Use case brief:

• We need to collect info from a lot of google ads accounts (at this time, 320 accounts)
• Some accounts got a kind of info that it will use to create reports.
• Some reports are create using a "join" process applying the info from differens accounts.
• We are generating 27 reports ...

This use case, running daily, it spends 13 hours to finish. running in a simple task. Slow networking, slow writing temp files, etc..

POC to do:

We need to launch some group of task running parallel. applying the same code using parameters (accounts id) Some task( or thinking as account google ads entity, can be relaunched after a "upgrade info" acoount process) Then, we expect, running in parallel to reduce time to finish , launching:

• task , collecting info from accounts ... processing parallel ... some code, different account.
• task join info from before and create reports

Then, some task could be restarted ... or not ... And, another corner case:

• Can i launch a task setting a "time stamp" sleep to exec ?
• For example, if account X failed to return all info, we need sleep 2 hours and retry .. next time, 6 hours ... can we do it applying task ?

Tx