Closed jeslinmx closed 1 year ago
The slow Python-language model.py
file is only used when sgp4
failed to compile the actual C++ SGP4 implementation when it installed on your machine. And since the SGP4 implementation cannot be vectorized (at least not without being fairly completely rewritten), both the Python and C++ versions of the code do call the underlying sgp4()
routine in a loop; for Python it's, of course, a slow loop, and in C it's a very fast one.
I will make two improvements to the documentation to resolve this issue:
print(sgp4.api.accelerated)
to see whether the C++ compiled and will make computations faster. I suggest you try running that, and sharing the results here on this issue, so that we can confirm that you don't have the C++ module available.Thanks for bringing this problem with the docs to my attention!
If you can find the errors that pip
encountered when installing sgp4
, we might be able to track down why you don't have the C++ compiled. Maybe you can find the pip
install logs still sitting around?
@jeslinmx — Good morning! I wanted to check back and find out whether you were able to confirm that the library is indeed running in Python-only mode on your machine?
Hey @brandon-rhodes, so sorry for the lack of response! I managed to get acceleration working earlier shortly after your first response. Long story short, I had installed skyfield through conda which also pulled in sgp4 as a dependency, and although skyfield itself was up to date (1.45), conda had decided that an ancient version of sgp4 (2.10) was sufficient. I was tipped off to this when I realized that sgp4.api.accelerated
didn't even exist in my install. Pulling the latest version of sgp4 directly through pip gave me working acceleration, and the runtime of sgp4_array is now down to ~17s from ~1300s!
So the original issue has been resolved, thank you again for your help and in following up!
Addendum: getting acceleration working through pip install
I wanted to recreate my environment using venv and pip rather than conda, but didn't have the bandwidth to do so (that's what delayed my response until now). And when attempting this, I found that installing sgp4 through pip doesn't provide acceleration, at least now out of the box.
Python 3.11.0 | packaged by conda-forge | (main, Jan 16 2023, 14:12:30) [MSC v.1916 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import sgp4.api
>>> sgp4.api.accelerated
False
(the "packaged by conda-forge" message is due to me installing Python 3.11 through conda, and then using it to create a venv)
Since I just recreated this environment, I have the logs right in front of me; they show no errors:
> pip install -r .\requirements.txt
Collecting pandas>=1.5
Using cached pandas-1.5.3-cp311-cp311-win_amd64.whl (10.3 MB)
Collecting skyfield>=1.45
Using cached skyfield-1.45-py3-none-any.whl (442 kB)
Collecting sgp4>=2.21
Using cached sgp4-2.21-cp311-cp311-win_amd64.whl
Collecting python-dateutil>=2.8.1
Using cached python_dateutil-2.8.2-py2.py3-none-any.whl (247 kB)
Collecting pytz>=2020.1
Using cached pytz-2022.7.1-py2.py3-none-any.whl (499 kB)
Collecting numpy>=1.21.0
Using cached numpy-1.24.2-cp311-cp311-win_amd64.whl (14.8 MB)
Collecting certifi>=2017.4.17
Using cached certifi-2022.12.7-py3-none-any.whl (155 kB)
Collecting jplephem>=2.13
Using cached jplephem-2.18-py3-none-any.whl (46 kB)
Collecting six>=1.5
Using cached six-1.16.0-py2.py3-none-any.whl (11 kB)
Installing collected packages: sgp4, pytz, six, numpy, certifi, python-dateutil, jplephem, skyfield, pandas
Successfully installed certifi-2022.12.7 jplephem-2.18 numpy-1.24.2 pandas-1.5.3 python-dateutil-2.8.2 pytz-2022.7.1 sgp4-2.21 six-1.16.0 skyfield-1.45
where requirements.txt
is nothing fancy
pandas>=1.5
openpyxl>=3.1
skyfield>=1.45
sgp4>=2.21
These are the contents of my venv:
> pip list
Package Version
----------------- ---------
asttokens 2.2.1
backcall 0.2.0
certifi 2022.12.7
colorama 0.4.6
comm 0.1.2
debugpy 1.6.6
decorator 5.1.1
et-xmlfile 1.1.0
executing 1.2.0
ipykernel 6.21.2
ipython 8.10.0
jedi 0.18.2
jplephem 2.18
jupyter_client 8.0.2
jupyter_core 5.2.0
matplotlib-inline 0.1.6
nest-asyncio 1.5.6
numpy 1.24.2
openpyxl 3.1.1
packaging 23.0
pandas 1.5.3
parso 0.8.3
pickleshare 0.7.5
pip 23.0
platformdirs 3.0.0
prompt-toolkit 3.0.36
psutil 5.9.4
pure-eval 0.2.2
Pygments 2.14.0
python-dateutil 2.8.2
pytz 2022.7.1
pywin32 305
pyzmq 25.0.0
setuptools 65.5.0
sgp4 2.21
six 1.16.0
skyfield 1.45
stack-data 0.6.2
tornado 6.2
traitlets 5.9.0
wcwidth 0.2.6
wheel 0.38.4
Acceleration works when sgp4 is installed through conda, these are the contents of my conda env:
> conda install --file .\skyfield-based\requirements.txt -c conda-forge
...
> conda list
# packages in environment at C:\Users\JeshuaLin\anaconda3\envs\speqtral1:
#
# Name Version Build Channel
asttokens 2.2.1 pyhd8ed1ab_0 conda-forge
backcall 0.2.0 pyh9f0ad1d_0 conda-forge
backports 1.0 pyhd8ed1ab_3 conda-forge
backports.functools_lru_cache 1.6.4 pyhd8ed1ab_0 conda-forge
bzip2 1.0.8 he774522_0
ca-certificates 2022.12.7 h5b45459_0 conda-forge
certifi 2022.12.7 pyhd8ed1ab_0 conda-forge
colorama 0.4.6 pyhd8ed1ab_0 conda-forge
comm 0.1.2 pyhd8ed1ab_0 conda-forge
debugpy 1.6.6 py311h12c1d0e_0 conda-forge
decorator 5.1.1 pyhd8ed1ab_0 conda-forge
et_xmlfile 1.1.0 pyhd8ed1ab_0 conda-forge
executing 1.2.0 pyhd8ed1ab_0 conda-forge
importlib-metadata 6.0.0 pyha770c72_0 conda-forge
importlib_metadata 6.0.0 hd8ed1ab_0 conda-forge
intel-openmp 2023.0.0 h57928b3_25922 conda-forge
ipykernel 6.21.2 pyh025b116_0 conda-forge
ipython 8.10.0 pyh08f2357_0 conda-forge
jedi 0.18.2 pyhd8ed1ab_0 conda-forge
jplephem 2.18 pyh78acc04_0 conda-forge
jupyter_client 8.0.2 pyhd8ed1ab_0 conda-forge
jupyter_core 5.2.0 py311h1ea47a8_0 conda-forge
libblas 3.9.0 16_win64_mkl conda-forge
libcblas 3.9.0 16_win64_mkl conda-forge
libffi 3.4.2 hd77b12b_6
liblapack 3.9.0 16_win64_mkl conda-forge
libsodium 1.0.18 h8d14728_1 conda-forge
matplotlib-inline 0.1.6 pyhd8ed1ab_0 conda-forge
mkl 2022.1.0 h6a75c08_874 conda-forge
nest-asyncio 1.5.6 pyhd8ed1ab_0 conda-forge
numpy 1.24.2 py311h0b4df5a_0 conda-forge
openpyxl 3.1.0 py311ha68e1ae_0 conda-forge
openssl 1.1.1t hcfcfb64_0 conda-forge
packaging 23.0 pyhd8ed1ab_0 conda-forge
pandas 1.5.3 py311hf63dbb6_0 conda-forge
parso 0.8.3 pyhd8ed1ab_0 conda-forge
pickleshare 0.7.5 py_1003 conda-forge
pip 22.2.2 py311haa95532_0
platformdirs 3.0.0 pyhd8ed1ab_0 conda-forge
prompt-toolkit 3.0.36 pyha770c72_0 conda-forge
psutil 5.9.4 py311ha68e1ae_0 conda-forge
pure_eval 0.2.2 pyhd8ed1ab_0 conda-forge
pygments 2.14.0 pyhd8ed1ab_0 conda-forge
python 3.11.0 h966fe2a_2
python-dateutil 2.8.2 pyhd8ed1ab_0 conda-forge
python_abi 3.11 2_cp311 conda-forge
pytz 2022.7.1 pyhd8ed1ab_0 conda-forge
pywin32 304 py311h12c1d0e_2 conda-forge
pyzmq 25.0.0 py311h7b3f143_0 conda-forge
setuptools 65.5.0 py311haa95532_0
sgp4 2.21 py311h12feb9d_2 conda-forge
six 1.16.0 pyh6c4a22f_0 conda-forge
skyfield 1.45 pyh1a96a4e_0 conda-forge
sqlite 3.40.1 h2bbff1b_0
stack_data 0.6.2 pyhd8ed1ab_0 conda-forge
tbb 2021.7.0 h91493d7_0 conda-forge
tk 8.6.12 h2bbff1b_0
tornado 6.2 py311ha68e1ae_1 conda-forge
traitlets 5.9.0 pyhd8ed1ab_0 conda-forge
typing-extensions 4.4.0 hd8ed1ab_0 conda-forge
typing_extensions 4.4.0 pyha770c72_0 conda-forge
tzdata 2022g h04d1e81_0
ucrt 10.0.22621.0 h57928b3_0 conda-forge
vc 14.2 h21ff451_1
vs2015_runtime 14.34.31931 h4c5c07a_10 conda-forge
wcwidth 0.2.6 pyhd8ed1ab_0 conda-forge
wheel 0.37.1 pyhd3eb1b0_0
wincertstore 0.2 py311haa95532_0
xz 5.2.10 h8cc25b3_1
zeromq 4.3.4 h0e60522_1 conda-forge
zipp 3.13.0 pyhd8ed1ab_0 conda-forge
zlib 1.2.13 h8cc25b3_0
I suppose the pip package is conservative in pulling along optional dependencies needed for acceleration while the conda install is more comprehensive. Would it be possible to add a section to the documentation regarding installation, advising users on what else to install for acceleration (if there isn't one already)?
Yes, the documentation will need to be expanded to cover your case so that folks know how to resolve it! It looks like the most recent version of the library doesn't have 3.11 wheels, because 3.11 didn't exist yet when it was last released on PyPI:
https://pypi.org/project/sgp4/#files
Which—I think means that your cached 3.11 wheel must have been built there on your machine? Does pip have any documentation that would let you ask "where did that cached wheel come from" or "when was it compiled" or "how can it be removed and rebuilt" or "how can I access its build logs to see if an error message indicates a C++ module failed to compile"?
Not too sure about the logs, but I can reproduce the situation with a new conda env:
> conda create -n test python=3.11
...
> conda activate test
> pip list
Package Version
------------ ---------
certifi 2022.9.24
pip 22.2.2
setuptools 65.5.0
wheel 0.37.1
wincertstore 0.2
> pip install --no-cache sgp4
Collecting sgp4
Downloading sgp4-2.21.tar.gz (162 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 162.1/162.1 kB 3.2 MB/s eta 0:00:00
Preparing metadata (setup.py) ... done
Building wheels for collected packages: sgp4
Building wheel for sgp4 (setup.py) ... done
Created wheel for sgp4: filename=sgp4-2.21-cp311-cp311-win_amd64.whl size=131621 sha256=152d4b63f2349031427b9c26f7d231f5bf56d13d2478cd580f5a8952486f3c76
Stored in directory: C:\Users\JeshuaLin\AppData\Local\Temp\pip-ephem-wheel-cache-2lpw2n8e\wheels\a1\42\f0\9b18d280c73a11deb760cef14227c9a276c86ed44c1ef84a39
Successfully built sgp4
Installing collected packages: sgp4
Successfully installed sgp4-2.21
> python -c "import sgp4.api; print(sgp4.api.accelerated)"
False
Build logs from rerunning the above with pip install -vvv
:
The key line is warning: build_ext: building extension "sgp4.vallado_cpp" failed: Microsoft Visual C++ 14.0 or greater is required. Get it with "Microsoft C++ Build Tools": https://visualstudio.microsoft.com/visual-cpp-build-tools/
, I guess. Which is true, I don't do any C++ dev on my Windows box (and I didn't expect to be compiling sgp4 today), so I guess that closes this case: lack of a 3.11 wheel in pypi and lack of a compiler on my part.
Running setup.py (path:C:\Users\JeshuaLin\AppData\Local\Temp\pip-install-9mox3xds\sgp4_0b50fc5d88ba46a3ba28325e20b80905\setup.py) egg_info for package sgp4
Created temporary directory: C:\Users\JeshuaLin\AppData\Local\Temp\pip-pip-egg-info-rvgqcyfm
Running command python setup.py egg_info
running egg_info
creating C:\Users\JeshuaLin\AppData\Local\Temp\pip-pip-egg-info-rvgqcyfm\sgp4.egg-info
writing C:\Users\JeshuaLin\AppData\Local\Temp\pip-pip-egg-info-rvgqcyfm\sgp4.egg-info\PKG-INFO
writing dependency_links to C:\Users\JeshuaLin\AppData\Local\Temp\pip-pip-egg-info-rvgqcyfm\sgp4.egg-info\dependency_links.txt
writing top-level names to C:\Users\JeshuaLin\AppData\Local\Temp\pip-pip-egg-info-rvgqcyfm\sgp4.egg-info\top_level.txt
writing manifest file 'C:\Users\JeshuaLin\AppData\Local\Temp\pip-pip-egg-info-rvgqcyfm\sgp4.egg-info\SOURCES.txt'
reading manifest file 'C:\Users\JeshuaLin\AppData\Local\Temp\pip-pip-egg-info-rvgqcyfm\sgp4.egg-info\SOURCES.txt'
adding license file 'LICENSE'
writing manifest file 'C:\Users\JeshuaLin\AppData\Local\Temp\pip-pip-egg-info-rvgqcyfm\sgp4.egg-info\SOURCES.txt'
Preparing metadata (setup.py) ... done
Source in c:\users\jeshualin\appdata\local\temp\pip-install-9mox3xds\sgp4_0b50fc5d88ba46a3ba28325e20b80905 has version 2.21, which satisfies requirement sgp4 from https://files.pythonhosted.org/packages/0e/20/a28ed2ffd30dcb3f090f636b5d110ffd148e8d26238987b3cdfaea6501b7/sgp4-2.21.tar.gz
Removed sgp4 from https://files.pythonhosted.org/packages/0e/20/a28ed2ffd30dcb3f090f636b5d110ffd148e8d26238987b3cdfaea6501b7/sgp4-2.21.tar.gz from build tracker 'C:\\Users\\JeshuaLin\\AppData\\Local\\Temp\\pip-build-tracker-f8fhr72u'
Created temporary directory: C:\Users\JeshuaLin\AppData\Local\Temp\pip-unpack-ih_ffmri
Building wheels for collected packages: sgp4
Created temporary directory: C:\Users\JeshuaLin\AppData\Local\Temp\pip-wheel-2qc98ihh
Building wheel for sgp4 (setup.py) ... Destination directory: C:\Users\JeshuaLin\AppData\Local\Temp\pip-wheel-2qc98ihh
Running command python setup.py bdist_wheel
running bdist_wheel
running build
running build_py
creating build
creating build\lib.win-amd64-cpython-311
creating build\lib.win-amd64-cpython-311\sgp4
copying sgp4\api.py -> build\lib.win-amd64-cpython-311\sgp4
copying sgp4\conveniences.py -> build\lib.win-amd64-cpython-311\sgp4
copying sgp4\earth_gravity.py -> build\lib.win-amd64-cpython-311\sgp4
copying sgp4\exporter.py -> build\lib.win-amd64-cpython-311\sgp4
copying sgp4\ext.py -> build\lib.win-amd64-cpython-311\sgp4
copying sgp4\functions.py -> build\lib.win-amd64-cpython-311\sgp4
copying sgp4\io.py -> build\lib.win-amd64-cpython-311\sgp4
copying sgp4\model.py -> build\lib.win-amd64-cpython-311\sgp4
copying sgp4\omm.py -> build\lib.win-amd64-cpython-311\sgp4
copying sgp4\propagation.py -> build\lib.win-amd64-cpython-311\sgp4
copying sgp4\tests.py -> build\lib.win-amd64-cpython-311\sgp4
copying sgp4\wrapper.py -> build\lib.win-amd64-cpython-311\sgp4
copying sgp4\wulfgar.py -> build\lib.win-amd64-cpython-311\sgp4
copying sgp4\__init__.py -> build\lib.win-amd64-cpython-311\sgp4
copying sgp4\SGP4-VER.TLE -> build\lib.win-amd64-cpython-311\sgp4
copying sgp4\sample_omm.csv -> build\lib.win-amd64-cpython-311\sgp4
copying sgp4\sample_omm.xml -> build\lib.win-amd64-cpython-311\sgp4
copying sgp4\tcppver.out -> build\lib.win-amd64-cpython-311\sgp4
running build_ext
building 'sgp4.vallado_cpp' extension
warning: build_ext: building extension "sgp4.vallado_cpp" failed: Microsoft Visual C++ 14.0 or greater is required. Get it with "Microsoft C++ Build Tools": https://visualstudio.microsoft.com/visual-cpp-build-tools/
C:\Users\JeshuaLin\anaconda3\envs\test\Lib\site-packages\setuptools\command\install.py:34: SetuptoolsDeprecationWarning: setup.py install is deprecated. Use build and pip and other standards-based tools.
warnings.warn(
installing to build\bdist.win-amd64\wheel
running install
running install_lib
creating build\bdist.win-amd64
creating build\bdist.win-amd64\wheel
creating build\bdist.win-amd64\wheel\sgp4
copying build\lib.win-amd64-cpython-311\sgp4\api.py -> build\bdist.win-amd64\wheel\.\sgp4
copying build\lib.win-amd64-cpython-311\sgp4\conveniences.py -> build\bdist.win-amd64\wheel\.\sgp4
copying build\lib.win-amd64-cpython-311\sgp4\earth_gravity.py -> build\bdist.win-amd64\wheel\.\sgp4
copying build\lib.win-amd64-cpython-311\sgp4\exporter.py -> build\bdist.win-amd64\wheel\.\sgp4
copying build\lib.win-amd64-cpython-311\sgp4\ext.py -> build\bdist.win-amd64\wheel\.\sgp4
copying build\lib.win-amd64-cpython-311\sgp4\functions.py -> build\bdist.win-amd64\wheel\.\sgp4
copying build\lib.win-amd64-cpython-311\sgp4\io.py -> build\bdist.win-amd64\wheel\.\sgp4
copying build\lib.win-amd64-cpython-311\sgp4\model.py -> build\bdist.win-amd64\wheel\.\sgp4
copying build\lib.win-amd64-cpython-311\sgp4\omm.py -> build\bdist.win-amd64\wheel\.\sgp4
copying build\lib.win-amd64-cpython-311\sgp4\propagation.py -> build\bdist.win-amd64\wheel\.\sgp4
copying build\lib.win-amd64-cpython-311\sgp4\sample_omm.csv -> build\bdist.win-amd64\wheel\.\sgp4
copying build\lib.win-amd64-cpython-311\sgp4\sample_omm.xml -> build\bdist.win-amd64\wheel\.\sgp4
copying build\lib.win-amd64-cpython-311\sgp4\SGP4-VER.TLE -> build\bdist.win-amd64\wheel\.\sgp4
copying build\lib.win-amd64-cpython-311\sgp4\tcppver.out -> build\bdist.win-amd64\wheel\.\sgp4
copying build\lib.win-amd64-cpython-311\sgp4\tests.py -> build\bdist.win-amd64\wheel\.\sgp4
copying build\lib.win-amd64-cpython-311\sgp4\wrapper.py -> build\bdist.win-amd64\wheel\.\sgp4
copying build\lib.win-amd64-cpython-311\sgp4\wulfgar.py -> build\bdist.win-amd64\wheel\.\sgp4
copying build\lib.win-amd64-cpython-311\sgp4\__init__.py -> build\bdist.win-amd64\wheel\.\sgp4
running install_egg_info
running egg_info
creating sgp4.egg-info
writing sgp4.egg-info\PKG-INFO
writing dependency_links to sgp4.egg-info\dependency_links.txt
writing top-level names to sgp4.egg-info\top_level.txt
writing manifest file 'sgp4.egg-info\SOURCES.txt'
reading manifest file 'sgp4.egg-info\SOURCES.txt'
adding license file 'LICENSE'
writing manifest file 'sgp4.egg-info\SOURCES.txt'
Copying sgp4.egg-info to build\bdist.win-amd64\wheel\.\sgp4-2.21-py3.11.egg-info
running install_scripts
C:\Users\JeshuaLin\anaconda3\envs\test\Lib\site-packages\wheel\bdist_wheel.py:80: RuntimeWarning: Config variable 'Py_DEBUG' is unset, Python ABI tag may be incorrect
if get_flag('Py_DEBUG',
adding license file "LICENSE" (matched pattern "LICEN[CS]E*")
creating build\bdist.win-amd64\wheel\sgp4-2.21.dist-info\WHEEL
creating 'C:\Users\JeshuaLin\AppData\Local\Temp\pip-wheel-2qc98ihh\sgp4-2.21-cp311-cp311-win_amd64.whl' and adding 'build\bdist.win-amd64\wheel' to it
adding 'sgp4/SGP4-VER.TLE'
adding 'sgp4/__init__.py'
adding 'sgp4/api.py'
adding 'sgp4/conveniences.py'
adding 'sgp4/earth_gravity.py'
adding 'sgp4/exporter.py'
adding 'sgp4/ext.py'
adding 'sgp4/functions.py'
adding 'sgp4/io.py'
adding 'sgp4/model.py'
adding 'sgp4/omm.py'
adding 'sgp4/propagation.py'
adding 'sgp4/sample_omm.csv'
adding 'sgp4/sample_omm.xml'
adding 'sgp4/tcppver.out'
adding 'sgp4/tests.py'
adding 'sgp4/wrapper.py'
adding 'sgp4/wulfgar.py'
adding 'sgp4-2.21.dist-info/LICENSE'
adding 'sgp4-2.21.dist-info/METADATA'
adding 'sgp4-2.21.dist-info/WHEEL'
adding 'sgp4-2.21.dist-info/top_level.txt'
adding 'sgp4-2.21.dist-info/RECORD'
removing build\bdist.win-amd64\wheel
done
Created wheel for sgp4: filename=sgp4-2.21-cp311-cp311-win_amd64.whl size=131621 sha256=ed4a7c5e3f6b6dcf7e676ab313673818de3c80500577b694cf9f6dc008d410f8
Stored in directory: C:\Users\JeshuaLin\AppData\Local\Temp\pip-ephem-wheel-cache-i2ms_dz2\wheels\a1\42\f0\9b18d280c73a11deb760cef14227c9a276c86ed44c1ef84a39
Successfully built sgp4
Installing collected packages: sgp4
Excellent, I am glad we have tracked down the exact cause of your trouble! I will plan to update the documentation to help people understand why acceleration might not be present, and then do a new release, which will not only update the online documentation but will also hopefully build some 3.11 wheels. I'll update this issue when I do!
Thank you for your efforts!
I've added the documentation:
f7cb6d6fff13d60b133f7ba539fc84a95b807b43
In a few days it should appear on PyPI, once I have the chance to do a release. Thanks for suggesting this improvment!
Let me first start off by saying that I love this library and Skyfield which it supports; it's been an incredibly useful and essential tool in my work.
I noted that in the documentation, the stated purpose of
Satrec.sgp4_array
andSatrecArray.sgp4
is "to avoid the expense of Python loops when you have many" dates and/or satellites. Accordingly, one would expect that the array of dates to be handled in a vectorized manner, a la Numpy.However, it does not seems that these 2 methods are just wrapping
Satrec.sgp4
in a python loop in the backend: https://github.com/brandon-rhodes/python-sgp4/blob/8f44082e3b16cc40f3a170b5ed819919a334c6f2/sgp4/model.py#L129-L131 https://github.com/brandon-rhodes/python-sgp4/blob/8f44082e3b16cc40f3a170b5ed819919a334c6f2/sgp4/model.py#L169-L172This also matches with what I observe in SnakeViz when using Skyfield (find_events of multiple satellites[^1], FWIW):
(note how 767 calls of
sgp4_array
results in 36658755 calls ofsgp4
)Is this the intended behavior, or a work in progress? Or am I misunderstanding the issue/incorrectly using Skyfield?
[^1]: on that note, is there possible to declare multiple satellites (say, a constellation) as a single object in Skyfield, and have their propagation handled in the backend by a
SatrecArray
rather than individualSatrec
s?