Open falkamelung opened 3 years ago
Hi!
I'm not sure if I understood correctly your problem, so sorry if not.
Stampede's CPU Intel(R) Xeon(R) Platinum 8160 CPU @ 2.10GHz has a total of 24 cores and 48 threads per processor (you can check it here) if you have two processors you have 96 total threads, which is what is reporting linux under /proc/cpuinfo, and dask is reporting 96threads which seems to be fine.
Yes, it reports 96 processors. I am not sure whether it should report this number but ifgram_inversion.py fails. In contrast, if I say mintpy.compute.numWorker=48
it works fine. I forgot to mention this, sorry.
Im confused. lscpu
says there are 96 CPUs, which is what the cpu_count finds. Thats what it supposed to be. Dask doesn't operate on "cores" it operates on "cpus" which are different things. There is no easy way to detect the number of cores as you defined them at runtime.
@falkamelung It'll be much simpler for you to just set minty.compute.numWorker to 48 and not touch it anymore. Additional searches have turned up anything that will pull just the number of cores instead of the cpu count.
Hi @Ovec8hkin , are you saying you did not find a command to retrieve the 'Thread(s) per core' from the system? As I said, num_core = os.cpu_count() / num_cpu_threads
would work. The issue is that we use the same mintpy installation on different queues with different hardware (different number of cores and threads) i.e. on nodes with different numbers of cores.
We could just run os.system('lscpu') but something pythonic would be preferred. If there is no immediate solution let me first try a few more systems to see whether os.system('lscpu') would indeed solve the problem.
There is no way to get the values you want in pure python. You can get either the total number of CPUs (96) via psutil.cpu_count()
or the number of cores per socket (24) via psutil.cpu_count(logical=False)
.
Parsing 'lscpu' via os
should not be done, as it resource intensive.
@yunjunz Thoughts on this? I would really hesitate at parsing the output of lscpu
in python looking for the proper values we want, as its a pretty limited use case, and I can't find any other means by which to get the number of cores Falk claims are available.
Are you sure the os.system('lscpu')
is a problem? It would be done only once in both ifgram_inversion
and dem_error
Using lscpu
sounds fine with me, since it only affect the translation of all
. Of course, we should explain it well in the comments of the function in cluster.py
and next to the option in smallbaselineApp.cfg
file.
Using os.system('lscpu')
is a messy work around for this, and doing things like that are generally not recommended by the python community, as they're comparatively expensive to using a native python command or full bash script. If @yunjunz doesn't mind, I guess you can go ahead and implement.
Send me a code block that pulls the correct data from the lscpu
output, and I will add it in the right spot and document it.
Hi @falkamelung and @Ovec8hkin, I have implemented num_core = os.cpu_count() / num_cpu_threads
as the translation for numWorker = all
, as Falk suggested. Cheers.
Thank you. Just to let you know dask has still a problem. It normally does not work for me to use numWorker= all
or numWorker=48
on stampede. I get weired dask connection errors. I normally use numWorker=36
which is a good compromise between failure frequency and speed. It would be good to find out why it fails if too many workers are given. I could provide examples.
@falkamelung Could it be that you run out of memory when you have many workers? Any idea how much memory you have on that machine and how much mintpy is using with 36 or 48 workers?
I don't think it is a memory problem. Here the error message that I get when I run a small dataset with 48 numWorkers. This dataset works fine with numWorkers=6. Below the output from the free
command.
cat smallbaseline_wrapper_8740432.e
ls: cannot access mintpy/timeseries*: No such file or directory
tornado.application - ERROR - Exception in callback functools.partial(<bound method IOLoop._discard_future_result of <tornado.platform.asyncio.AsyncIOLoop object at 0x2ab8e67b5970>>, <Task finished name='Task-947' coro=<Cluster._sync_cluster_info() done, defined at /work2/05861/tg851601/stampede2/code/rsmas_insar/3rdparty/miniconda3/lib/python3.8/site-packages/distributed/deploy/cluster.py:104> exception=OSError('Timed out during handshake while connecting to tcp://127.0.0.1:43192 after 30 s')>)
Traceback (most recent call last):
File "/work2/05861/tg851601/stampede2/code/rsmas_insar/3rdparty/miniconda3/lib/python3.8/site-packages/distributed/comm/core.py", line 319, in connect
handshake = await asyncio.wait_for(comm.read(), time_left())
File "/tmp/rsmas_insar/3rdparty/miniconda3/lib/python3.8/asyncio/tasks.py", line 464, in wait_for
raise exceptions.TimeoutError()
asyncio.exceptions.TimeoutError
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/work2/05861/tg851601/stampede2/code/rsmas_insar/3rdparty/miniconda3/lib/python3.8/site-packages/tornado/ioloop.py", line 741, in _run_callback
ret = callback()
File "/work2/05861/tg851601/stampede2/code/rsmas_insar/3rdparty/miniconda3/lib/python3.8/site-packages/tornado/ioloop.py", line 765, in _discard_future_result
future.result()
File "/work2/05861/tg851601/stampede2/code/rsmas_insar/3rdparty/miniconda3/lib/python3.8/site-packages/distributed/deploy/cluster.py", line 105, in _sync_cluster_info
await self.scheduler_comm.set_metadata(
File "/work2/05861/tg851601/stampede2/code/rsmas_insar/3rdparty/miniconda3/lib/python3.8/site-packages/distributed/core.py", line 785, in send_recv_from_rpc
comm = await self.live_comm()
File "/work2/05861/tg851601/stampede2/code/rsmas_insar/3rdparty/miniconda3/lib/python3.8/site-packages/distributed/core.py", line 742, in live_comm
comm = await connect(
File "/work2/05861/tg851601/stampede2/code/rsmas_insar/3rdparty/miniconda3/lib/python3.8/site-packages/distributed/comm/core.py", line 324, in connect
raise OSError(
OSError: Timed out during handshake while connecting to tcp://127.0.0.1:43192 after 30 s
tornado.application - ERROR - Exception in callback functools.partial(<bound method IOLoop._discard_future_result of <tornado.platform.asyncio.AsyncIOLoop object at 0x2ab8e67b5970>>, <Task finished name='Task-952' coro=<Cluster._sync_cluster_info() done, defined at /work2/05861/tg851601/stampede2/code/rsmas_insar/3rdparty/miniconda3/lib/python3.8/site-packages/distributed/deploy/cluster.py:104> exception=OSError('Timed out during handshake while connecting to tcp://127.0.0.1:43192 after 30 s')>)
Traceback (most recent call last):
File "/work2/05861/tg851601/stampede2/code/rsmas_insar/3rdparty/miniconda3/lib/python3.8/site-packages/distributed/comm/core.py", line 319, in connect
handshake = await asyncio.wait_for(comm.read(), time_left())
File "/tmp/rsmas_insar/3rdparty/miniconda3/lib/python3.8/asyncio/tasks.py", line 464, in wait_for
raise exceptions.TimeoutError()
asyncio.exceptions.TimeoutError
free -h
total used free shared buff/cache available
Mem: 187G 10G 164G 349M 12G 172G
Swap: 0B 0B 0B
The Linux server I was using recently has 16 cores and two threads per core. It ran successfully with 32 parallel dask processes. It was using files on a ZFS file system that is directly connected. I did not check how much memory was used, but the machine has 128 GB of RAM.
Thank you Falk and Eric for the info. This means we still have not located the cause yet. The "conflict during multiple HDF5 writing processes" as described in #692 still sounds like a reasonable guess to me.
Update: multiple HDF5 writing processes do not exist in mintpy, thus, that is not the cause.
We may revert the num_core = os.cpu_count() / num_cpu_threads
translation in the future once the issue is cleared.
All the other errors, e.g. no access to file
, negative dimension size
, have been fixed by the PR above. The remaining re-occurring error is OSError: Timed out during handshake while connecting to tcp
from dask. A quick search and try did not fix this issue completely on stampede.
To better handle this situation, I added the support of mintpy.compute.numWorker = 75%
style input in the linked PR above.
Description of the problem I am running
ifgram_inversion.py
usingmintpy.compute.numWorker=all
on stampede2 (48 cores per node, each 2 threads) and it thinks there are 96 cores:This value is returned by the python
num_core = os.cpu_count()
command inmintpy/objects/cluster.py
. So we need another command to figure out the number of threads and divide. Thelscpu
command does return the proper amount of cores and threads on both Stampede2 and Frontera. I would expect that there is an equivalent in python.Stampede:
Frontera: