make small case for azure test failing for cogent3

[GavinHuttley]

the azure test is failing for the following cogent3 test, which indicates we might need to be able to specify a container with more than 1 process.

can you modify your demo scitrack azure pipeline case to add this one test and see if you can establish a config that allows this to pass?

   def test_create_processes(self):
        """Procressor pool should create multiple distingue processes"""
        index = [2, 3, 4, 5, 6, 7, 8, 9, 10]
>       result = parallel.map(get_process_value, index, max_workers=2, use_mpi=False)

test_util/test_parallel.py:39: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
../.tox/py37/lib/python3.7/site-packages/cogent3/util/parallel.py:168: in map
    return list(imap(f, s, max_workers, use_mpi, if_serial, chunksize))
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

f = <function get_process_value at 0x7efcc01a7320>, s = [2, 3, 4, 5, 6, 7, ...]
max_workers = 2, use_mpi = False, if_serial = 'raise', chunksize = None

    def imap(f, s, max_workers=None, use_mpi=False, if_serial="raise", chunksize=None):
        """
        Parameters
        ----------
        f : callable
            function that operates on values in s
        s : iterable
            series of inputs to f
        max_workers : int or None
            maximum number of workers. Defaults to 1-maximum available.
        use_mpi : bool
            use MPI for parallel execution
        if_serial : str
            action to take if conditions will result in serial execution. Valid
            values are 'raise', 'ignore', 'warn'. Defaults to 'raise'.
        chunksize : int or None
            Size of data chunks executed by worker processes. Defaults to None
            where stable chunksize is determined by set_default_chunksize()

        Returns
        -------
        imap is a generator yielding result of f(s[i]), map returns the result
        series
        """

        if_serial = if_serial.lower()
        assert if_serial in ("ignore", "raise", "warn"), f"invalid choice '{if_serial}'"

        # If max_workers is not defined, get number of all processes available
        # minus 1 to leave for master process
        if use_mpi:
            if not USING_MPI:
                raise RuntimeError("Cannot use MPI")

            err_msg = (
                "Execution in serial. For parallel MPI execution, use:\n"
                " $ mpirun -n 1 <executable script>"
            )

            if COMM.Get_attr(MPI.UNIVERSE_SIZE) == 1 and if_serial == "raise":
                raise RuntimeError(err_msg)
            elif COMM.Get_attr(MPI.UNIVERSE_SIZE) == 1 and if_serial == "warn":
                warnings.warn(UserWarning, msg=err_msg)

            if not max_workers:
                max_workers = COMM.Get_attr(MPI.UNIVERSE_SIZE) - 1

            if not chunksize:
                chunksize = set_default_chunksize(s, max_workers)

            with MPIfutures.MPIPoolExecutor(max_workers=max_workers) as executor:
                for result in executor.map(f, s, chunksize=chunksize):
                    yield result
        else:
            if not max_workers:
                max_workers = multiprocessing.cpu_count() - 1
>           assert max_workers < multiprocessing.cpu_count()
E           AssertionError

[thomas-la]

thomas@Azure:~$ lscpu
Architecture:          x86_64
CPU op-mode(s):        32-bit, 64-bit
Byte Order:            Little Endian
CPU(s):                2
On-line CPU(s) list:   0,1
Thread(s) per core:    1
Core(s) per socket:    2
Socket(s):             1
NUMA node(s):          1
Vendor ID:             GenuineIntel
CPU family:            6
Model:                 63
Model name:            Intel(R) Xeon(R) CPU E5-2673 v3 @ 2.40GHz
Stepping:              2
CPU MHz:               2397.225
BogoMIPS:              4794.45
Hypervisor vendor:     Microsoft
Virtualization type:   full
L1d cache:             32K
L1i cache:             32K
L2 cache:              256K
L3 cache:              30720K
NUMA node0 CPU(s):     0,1
Flags:                 fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss ht syscall nx pdpe1gb rdtscp lm constant_tsc rep_good nopl xtopology cpuid pnipclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand hypervisor lahf_lm abm invpcid_single pti fsgsbase bmi1 avx2 smep bmi2 erms invpcid xsaveopt md_clear```

[thomas-la]

After consultation with Sheng, running parallel on this system should give only 1 worker node...

[thomas-la]

Moses's PR (cogent3 #217 for #215) addressed this issue.

[GavinHuttley]

no longer using azure