eBay / parallec

Fast Parallel Async HTTP/SSH/TCP/UDP/Ping Client Java Library. Aggregate 100,000 APIs & send anywhere in 20 lines of code. Ping/HTTP Calls 8000 servers in 12 seconds. (Akka) www.parallec.io
Apache License 2.0
805 stars 173 forks source link

How to create task with m2m request #71

Open csliangdu opened 6 years ago

csliangdu commented 6 years ago

I have two servers, A and B. For server A, i got 3 requests, i.e., a1, a2, a3. Similarly, i got 3 request on server B, i.e., b1, b2, b3. Totally, i have 6 requests on 2 servers, each server with 3 requests.

In the example code, "HttpDiffRequestsDiffServersApp.java" and "HttpDiffRequestsSameServerApp.java". it has been show how to start requests {A.a1, B.b1} and {A.a1, A.a2, A.a3}.

Furthermore, in "ParallelTaskBuilder.setReplacementVarMapNodeSpecific()", it has "this.replacementVarMapNodeSpecific.clear();". Such code EXCLUDE the requests from other commands.

Is there any way to start requests {A.a1, A.a2, A.a3, B.b1, B.b2, B.b3} in one job?

Liang

jeffpeiyt commented 6 years ago

@csliangdu good question. I believe by looking at the implementation of the DiffRequestsSameServer we can do sth works for this.

Will be pretty busy this week and can dig deeper this weekend, if not earlier...

csliangdu commented 6 years ago

Thanks for your quick response.

By adding a new method setReplaceVarMapToMultipleTarget() in ParallelTaskBuilder, it seems the problem of building complex requests could be handled.

    public ParallelTaskBuilder setReplaceVarMapToMultipleTarget(
            String variable, List<List<String>> replaceLists, List<String> targetHosts ){
        if (replaceLists.size() != targetHosts.size()){
            logger.error("<replaceLists, targetHosts> should be same size ...");
            return this;
        }
        Map<String, StrStrMap> replacementVarMapNodeSpecificAll = new HashMap<String, StrStrMap>();
        List<String> targetHostsAll = new ArrayList<String>();
        for (int i = 0; i < replaceLists.size(); i++){
            setReplaceVarMapToSingleTargetSingleVar(variable, replaceLists.get(i), targetHosts.get(i));         
            for (Entry<String, StrStrMap> entry : this.replacementVarMapNodeSpecific
                    .entrySet()) {
                replacementVarMapNodeSpecificAll.put(entry.getKey() + "_" + i, entry.getValue());// with additional surfix
            }
            for (int j = 0; j < this.targetHosts.size(); j++){
                targetHostsAll.add(this.targetHosts.get(j) + "_" + i);// with additional surfix
            }
        }
        this.replacementVarMapNodeSpecific.clear();
        this.replacementVarMapNodeSpecific.putAll(replacementVarMapNodeSpecificAll);
        this.targetHosts.clear();
        this.targetHosts.addAll(targetHostsAll);        
        return this;
    }

Test

        ParallelTaskBuilder ptb = pc.prepareHttpGet("/$JOB_ID");

        List<List<String>> replaceLists = new ArrayList<List<String>>();
        List<String> targetHosts = new ArrayList<String>();

        //handle requests from server A
        targetHosts.add("A");
        List<String> replaceList = new ArrayList<String>();
        replaceList.add("a1");
        replaceList.add("a2");
        replaceList.add("a3");              
        replaceLists.add(replaceList);

        //handle requests from server B
        targetHosts.add("B");
        List<String> replaceList2 = new ArrayList<String>();
        replaceList2.add("b1");
        replaceList2.add("b2");
        replaceLists.add(replaceList2);

        ptb.setReplaceVarMapToMultipleTarget("JOB_ID", replaceLists, targetHosts);

However, I have no idea about its influence to the rest workflow.

Liang