search

antoinecarme / pyaf

PyAF is an Open Source Python library for Automatic Time Series Forecasting built on top of popular pydata modules.
BSD 3-Clause "New" or "Revised" License
458 stars 73 forks source link

Add some tests in a Massively Parallel Forecasting Architecture #115

Closed antoinecarme closed 4 years ago

antoinecarme commented 4 years ago

PyAF uses a parallel training process with 4 sub-processes by default , which is OK for a standard PC with about 8 cores. the number of these sub-processes is configurable.

https://github.com/antoinecarme/pyaf/blob/2bee2a65319a2f4582c5249c00c8f17856887654/TS/Options.py#L121

Need to see what happens and what can be improved when one has hunderds of Cores.

The Xeon-Phi Architecture, recently made EOL by Intel, is a good candidate for these tests. Each Xeon-Phi processor has at least 64 Xeon-like cores or 256 concurrent threads.

antoinecarme commented 4 years ago

setup a xeon-phi debian machine for the tests

New repository for debian-related config data :

https://github.com/antoinecarme/xeon-phi-data

antoinecarme commented 4 years ago

Hierarchical modeling is now parallelized differently. The number of cores needed is the number of nodes in the hierarchy, All individual node models are trained in parallel.

The same for forecasting. All individual node models are forecast in parallel.

antoinecarme commented 4 years ago

Closing.

Will be officially available in release 2.0