Make `rms-pdsfile` standalone

juzen2003 commented 8 months ago

Updates:

Updated requirements.in to include rms-pdstable, rms-pdslogger, rms-translator and pillow
Added pdsviewable.py under pdsfile and updated the import correspondingly
Remove pdsgroup related functions in the tests
Replace os.path.join with _clean_join in pdsfile.py

Setup notes:

Make sure nothing is set in local PYTHONPATH, no rms-pdsfile and rms-webtools
Go to local rms-pdsfile directory and set up virtual environement by doing python -m venv venv (or name your preferred venv folder)
Install packages by doing pip install -r requirements.txt, before doing this, if you try to do import pdsfile in ipython or do ./run_tests_coverage.sh in terminal it will fail because packages are not installed.
After above steps, ./run_tests_coverage.sh will pass, and import will work fine (similar to these in ipython):
```
In [1]: import pdsfile
```

In [3]: print(pdsfile.file) /Users/yu-jenchang/Desktop/seti/workspace/rms-pdsfile/pdsfile/init.py

In [4]: import pdslogger

In [5]: print(pdslogger.file) /Users/yu-jenchang/Desktop/seti/workspace/rms-pdsfile/venv/lib/python3.10/site-packages/pdslogger/init.py

In [6]: import pdstable

In [7]: print(pdstable.file) /Users/yu-jenchang/Desktop/seti/workspace/rms-pdsfile/venv/lib/python3.10/site-packages/pdstable/init.py

In [8]: import translator

In [9]: print(translator.file) /Users/yu-jenchang/Desktop/seti/workspace/rms-pdsfile/venv/lib/python3.10/site-packages/translator/init.py

In [11]: print(pdsfile.pdsviewable.file) /Users/yu-jenchang/Desktop/seti/workspace/rms-pdsfile/pdsfile/pdsviewable.py

In [12]: print(pdsfile.pdscache.file) /Users/yu-jenchang/Desktop/seti/workspace/rms-pdsfile/pdsfile/pdscache.py

rfrenchseti commented 8 months ago

I'm going to suggest we don't use pip-compile and requirements.in but just use normal requirements.txt. The reason is two-fold: 1) With requirements.in you end up freezing particular versions of every module, and having to manually update to newer versions. This shouldn't be necessary for a package like this. 2) rms-pdslogger on Linux and MacOS depends on xattr but on Windows it doesn't. Since the requirements.in file was processed on MacOS, xattr is in requirements.txt. Unfortunately that module can't be installed at all on Windows, which means you can't actually do pip install -r requirements.txt. This problem doesn't happen if we stick to the old way of doing things.

juzen2003 commented 7 months ago

Logs for reference:

create virtual env
activate virtual env
run tests but failed because we haven't installed packages yet
install packages

run tests and passed


yu-jenchang make_rms_pdsfile_standalone rms-pdsfile $ rm -rf venv                  
yu-jenchang make_rms_pdsfile_standalone rms-pdsfile $ python -m venv venv          
yu-jenchang make_rms_pdsfile_standalone rms-pdsfile $ source venv/bin/activate
(venv) yu-jenchang make_rms_pdsfile_standalone rms-pdsfile $ ./run_tests_coverage.sh 
Clean up previous coverage record
Run with use shelves
/opt/homebrew/lib/python3.10/site-packages/coverage/inorout.py:460: CoverageWarning: --include is ignored because --source is set (include-ignored)
self.warn("--include is ignored because --source is set", slug="include-ignored")
ImportError while loading conftest '/Users/yu-jenchang/Desktop/seti/workspace/rms-pdsfile/conftest.py'.
conftest.py:8: in <module>
from pdsfile import (Pds3File,
pdsfile/__init__.py:6: in <module>
from .pds3file import *
pdsfile/pds3file/__init__.py:6: in <module>
import pdslogger
E   ModuleNotFoundError: No module named 'pdslogger'
Run with no shelves
/opt/homebrew/lib/python3.10/site-packages/coverage/inorout.py:460: CoverageWarning: --include is ignored because --source is set (include-ignored)
self.warn("--include is ignored because --source is set", slug="include-ignored")
ImportError while loading conftest '/Users/yu-jenchang/Desktop/seti/workspace/rms-pdsfile/conftest.py'.
conftest.py:8: in <module>
from pdsfile import (Pds3File,
pdsfile/__init__.py:6: in <module>
from .pds3file import *
pdsfile/pds3file/__init__.py:6: in <module>
import pdslogger
E   ModuleNotFoundError: No module named 'pdslogger'
Combine results from all modes
Combined data file .coverage.YuJensMBP2023M2.72748.065746
Skipping duplicate data .coverage.YuJensMBP2023M2.72745.419978
Generate html
Wrote HTML report to htmlcov/index.html
Report coverage
Name                                                      Stmts   Miss Branch BrPart  Cover
-------------------------------------------------------------------------------------------
pdsfile/__init__.py                                           3      1      0      0    67%
pdsfile/general_helper.py                                    86     86     46      0     0%
pdsfile/pds3file/__init__.py                                 87     86      6      0     1%
pdsfile/pds3file/rules/ASTROM_xxxx.py                         8      8      0      0     0%
pdsfile/pds3file/rules/COCIRS_xxxx.py                        99     99     38      0     0%
pdsfile/pds3file/rules/COISS_xxxx.py                         73     73     26      0     0%
pdsfile/pds3file/rules/CORSS_8xxx.py                        126    126     60      0     0%
pdsfile/pds3file/rules/COSP_xxxx.py                          13     13      0      0     0%
pdsfile/pds3file/rules/COUVIS_0xxx.py                       100    100     40      0     0%
pdsfile/pds3file/rules/COUVIS_8xxx.py                        71     71     24      0     0%
pdsfile/pds3file/rules/COVIMS_0xxx.py                        98     98     44      0     0%
pdsfile/pds3file/rules/COVIMS_8xxx.py                        69     69     26      0     0%
pdsfile/pds3file/rules/EBROCC_xxxx.py                        73     73     30      0     0%
pdsfile/pds3file/rules/GO_0xxx.py                           128    128     46      0     0%
pdsfile/pds3file/rules/HSTxx_xxxx.py                         80     80     30      0     0%
pdsfile/pds3file/rules/JNOJIR_xxxx.py                        23     23      0      0     0%
pdsfile/pds3file/rules/JNOJNC_xxxx.py                        17     17      0      0     0%
pdsfile/pds3file/rules/JNOSP_xxxx.py                         13     13      0      0     0%
pdsfile/pds3file/rules/NHSP_xxxx.py                          13     13      0      0     0%
pdsfile/pds3file/rules/NHxxxx_xxxx.py                       120    120     52      0     0%
pdsfile/pds3file/rules/RES_xxxx.py                            6      6      0      0     0%
pdsfile/pds3file/rules/RPX_xxxx.py                           29     29      2      0     0%
pdsfile/pds3file/rules/VGIRIS_xxxx.py                        10     10      0      0     0%
pdsfile/pds3file/rules/VGISS_xxxx.py                         71     71     24      0     0%
pdsfile/pds3file/rules/VG_0xxx.py                            26     26      0      0     0%
pdsfile/pds3file/rules/VG_20xx.py                            10     10      0      0     0%
pdsfile/pds3file/rules/VG_28xx.py                            55     55      0      0     0%
pdsfile/pds3file/rules/__init__.py                           25     25      0      0     0%
pdsfile/pds3file/rules/pytest_support.py                     19     19      0      0     0%
pdsfile/pds3file/tests/__init__.py                            0      0      0      0   100%
pdsfile/pds3file/tests/helper.py                              9      9      0      0     0%
pdsfile/pds3file/tests/test_pds3file_blackbox.py            593    593     46      0     0%
pdsfile/pds3file/tests/test_pds3file_blackbox_cached.py     275    275     18      0     0%
pdsfile/pds3file/tests/test_pds3file_whitebox.py            301    301     36      0     0%
pdsfile/pds3file/tests/test_pdsviewable_blackbox.py          24     24      0      0     0%
pdsfile/pds4file/__init__.py                                 68     68      6      0     0%
pdsfile/pds4file/rules/__init__.py                           25     25      0      0     0%
pdsfile/pds4file/rules/cassini_iss.py                        75     75     26      0     0%
pdsfile/pds4file/rules/cassini_vims.py                       75     75     26      0     0%
pdsfile/pds4file/rules/pytest_support.py                     19     19      0      0     0%
pdsfile/pds4file/rules/uranus_occs_earthbased.py             75     75     26      0     0%
pdsfile/pds4file/tests/__init__.py                            0      0      0      0   100%
pdsfile/pds4file/tests/helper.py                              5      5      0      0     0%
pdsfile/pds4file/tests/test_pds4file_blackbox.py             84     84      4      0     0%
pdsfile/pdsfile.py                                         3003   3003   1474      0     0%
pdsfile/pdsviewable.py                                      269    269    124      0     0%
pdsfile/preload_and_cache.py                                 23     23     10      0     0%
-------------------------------------------------------------------------------------------
TOTAL                                                      6474   6471   2290      0     1%
(venv) yu-jenchang make_rms_pdsfile_standalone rms-pdsfile $ pip install -r requirements.txt 
Collecting coverage
Downloading coverage-7.4.0-cp39-cp39-macosx_11_0_arm64.whl (205 kB)
 |████████████████████████████████| 205 kB 2.8 MB/s 
Collecting pillow
Downloading pillow-10.2.0-cp39-cp39-macosx_11_0_arm64.whl (3.3 MB)
 |████████████████████████████████| 3.3 MB 13.0 MB/s 
Collecting pylibmc
Using cached pylibmc-1.6.3-cp39-cp39-macosx_11_0_arm64.whl (205 kB)
Collecting pytest
Downloading pytest-7.4.4-py3-none-any.whl (325 kB)
 |████████████████████████████████| 325 kB 9.0 MB/s 
Collecting rms-pdslogger
Downloading rms_pdslogger-0.0.5-py3-none-any.whl (14 kB)
Collecting rms-pdstable
Downloading rms_pdstable-0.0.5-py3-none-any.whl (17 kB)
Collecting rms-translator
Downloading rms_translator-0.0.1-py3-none-any.whl (9.5 kB)
Collecting exceptiongroup>=1.0.0rc8
Downloading exceptiongroup-1.2.0-py3-none-any.whl (16 kB)
Collecting iniconfig
Using cached iniconfig-2.0.0-py3-none-any.whl (5.9 kB)
Collecting tomli>=1.0.0
Using cached tomli-2.0.1-py3-none-any.whl (12 kB)
Collecting pluggy<2.0,>=0.12
Downloading pluggy-1.3.0-py3-none-any.whl (18 kB)
Collecting packaging
Using cached packaging-23.2-py3-none-any.whl (53 kB)
Collecting numpy
Downloading numpy-1.26.3-cp39-cp39-macosx_11_0_arm64.whl (14.0 MB)
 |████████████████████████████████| 14.0 MB 4.7 MB/s 
Collecting pyparsing
Downloading pyparsing-3.1.1-py3-none-any.whl (103 kB)
 |████████████████████████████████| 103 kB 10.8 MB/s 
Collecting rms-textkernel
Downloading rms_textkernel-1.0.0-py3-none-any.whl (42 kB)
 |████████████████████████████████| 42 kB 3.1 MB/s 
Collecting xattr
Downloading xattr-1.0.0-cp39-cp39-macosx_11_0_arm64.whl (18 kB)
Collecting rms-julian
Downloading rms_julian-2.0.0-py3-none-any.whl (74 kB)
 |████████████████████████████████| 74 kB 8.8 MB/s 
Collecting rms-pdsparser
Downloading rms_pdsparser-0.0.4-py3-none-any.whl (16 kB)
Collecting cffi>=1.16.0
Downloading cffi-1.16.0-cp39-cp39-macosx_11_0_arm64.whl (176 kB)
 |████████████████████████████████| 176 kB 10.9 MB/s 
Collecting pycparser
Downloading pycparser-2.21-py2.py3-none-any.whl (118 kB)
 |████████████████████████████████| 118 kB 7.6 MB/s 
Installing collected packages: pyparsing, pycparser, numpy, rms-textkernel, rms-julian, cffi, xattr, tomli, rms-pdsparser, pluggy, packaging, iniconfig, exceptiongroup, rms-translator, rms-pdstable, rms-pdslogger, pytest, pylibmc, pillow, coverage
Successfully installed cffi-1.16.0 coverage-7.4.0 exceptiongroup-1.2.0 iniconfig-2.0.0 numpy-1.26.3 packaging-23.2 pillow-10.2.0 pluggy-1.3.0 pycparser-2.21 pylibmc-1.6.3 pyparsing-3.1.1 pytest-7.4.4 rms-julian-2.0.0 rms-pdslogger-0.0.5 rms-pdsparser-0.0.4 rms-pdstable-0.0.5 rms-textkernel-1.0.0 rms-translator-0.0.1 tomli-2.0.1 xattr-1.0.0
(venv) yu-jenchang make_rms_pdsfile_standalone rms-pdsfile $ ./run_tests_coverage.sh        
Clean up previous coverage record
Run with use shelves
/Users/yu-jenchang/Desktop/seti/workspace/rms-pdsfile/venv/lib/python3.9/site-packages/coverage/inorout.py:456: CoverageWarning: --include is ignored because --source is set (include-ignored)
self.warn("--include is ignored because --source is set", slug="include-ignored")
============================================= test session starts =============================================
platform darwin -- Python 3.9.6, pytest-7.4.4, pluggy-1.3.0
rootdir: /Users/yu-jenchang/Desktop/seti/workspace/rms-pdsfile
collected 633 items

pdsfile/pds3file/tests/test_pds3file_blackbox.py ......................................... ......................................... ...................... pdsfile/pds3file/tests/test_pds3file_blac ................................................. pdsfile/pds3file/tests/test_pds3file_whitebox.py ..................................... pdsfile/pds3file/tests/test_pdsviewable_blackbox.py pdsfile/pds3file/rules/COCIRS_xxxx.py ..... pdsfile/pds3file/rules/COISS_xxxx.py .. pdsfile/pds3file/rules/CORSS_8xxx.py .... pdsfile/pds3file/rules/COUVIS_0xxx.py ........... pdsfile/pds3file/rules/COUVIS_8xxx.py ....... pdsfile/pds3file/rules/COVIMS_0xxx.py ... pdsfile/pds3file/rules/COVIMS_8xxx.py .. pdsfile/pds3file/rules/EBROCC_xxxx.py ... pdsfile/pds3file/rules/GO_0xxx.py ... pdsfile/pds3file/rules/HSTxx_xxxx.py ... pdsfile/pds3file/rules/NHxxxx_xxxx.py ... pdsfile/pds3file/rules/VGISS_xxxx.py .. pdsfile/pds3file/rules/VG_28xx.py ........ pdsfile/pds4file/tests/test_pds4file_blackbox.py .................................................. ...................................................... [ 8%] .............................................................. [ 24%] .............................................................. [ 41%] [ 44%] kbox_cached.py ............................................... [ 51%] [ 59%] ...................................................... [ 68%] [ 74%] .... [ 74%] [ 75%] [ 75%] [ 76%] [ 78%] [ 79%] [ 79%] [ 80%] [ 80%] [ 81%] [ 81%] [ 81%] [ 82%] [ 83%] ...................................................... [ 92%] [100%]

======================================= 633 passed in 109.97s (0:01:49) ======================================= Run with no shelves /Users/yu-jenchang/Desktop/seti/workspace/rms-pdsfile/venv/lib/python3.9/site-packages/coverage/inorout.py:456: CoverageWarning: --include is ignored because --source is set (include-ignored) self.warn("--include is ignored because --source is set", slug="include-ignored") ============================================= test session starts ============================================= platform darwin -- Python 3.9.6, pytest-7.4.4, pluggy-1.3.0 rootdir: /Users/yu-jenchang/Desktop/seti/workspace/rms-pdsfile collected 633 items

pdsfile/pds3file/tests/test_pds3file_blackbox.py ...................................................... [ 8%] ....................................................................................................... [ 24%] ....................................................................................................... [ 41%] ...................... [ 44%] pdsfile/pds3file/tests/test_pds3file_blackbox_cached.py ............................................... [ 51%] ................................................. [ 59%] pdsfile/pds3file/tests/test_pds3file_whitebox.py ...................................................... [ 68%] ..................................... [ 74%] pdsfile/pds3file/tests/test_pdsviewable_blackbox.py .... [ 74%] pdsfile/pds3file/rules/COCIRS_xxxx.py ..... [ 75%] pdsfile/pds3file/rules/COISS_xxxx.py .. [ 75%] pdsfile/pds3file/rules/CORSS_8xxx.py .... [ 76%] pdsfile/pds3file/rules/COUVIS_0xxx.py ........... [ 78%] pdsfile/pds3file/rules/COUVIS_8xxx.py ....... [ 79%] pdsfile/pds3file/rules/COVIMS_0xxx.py ... [ 79%] pdsfile/pds3file/rules/COVIMS_8xxx.py .. [ 80%] pdsfile/pds3file/rules/EBROCC_xxxx.py ... [ 80%] pdsfile/pds3file/rules/GO_0xxx.py ... [ 81%] pdsfile/pds3file/rules/HSTxx_xxxx.py ... [ 81%] pdsfile/pds3file/rules/NHxxxx_xxxx.py ... [ 81%] pdsfile/pds3file/rules/VGISS_xxxx.py .. [ 82%] pdsfile/pds3file/rules/VG_28xx.py ........ [ 83%] pdsfile/pds4file/tests/test_pds4file_blackbox.py ...................................................... [ 92%] .................................................. [100%]

======================================= 633 passed in 65.11s (0:01:05) ======================================== Combine results from all modes Combined data file .coverage.YuJensMBP2023M2.79128.XkqoHRDx Combined data file .coverage.YuJensMBP2023M2.76668.Xwlnqekx Generate html Wrote HTML report to htmlcov/index.html Report coverage Name Stmts Miss Branch BrPart Cover

pdsfile/init.py 3 0 0 0 100% pdsfile/general_helper.py 86 24 46 2 70% pdsfile/pds3file/init.py 87 8 6 1 88% pdsfile/pds3file/rules/ASTROM_xxxx.py 8 0 0 0 100% pdsfile/pds3file/rules/COCIRS_xxxx.py 99 4 38 2 93% pdsfile/pds3file/rules/COISS_xxxx.py 73 3 26 0 95% pdsfile/pds3file/rules/CORSS_8xxx.py 126 0 60 0 100% pdsfile/pds3file/rules/COSP_xxxx.py 13 0 0 0 100% pdsfile/pds3file/rules/COUVIS_0xxx.py 100 4 40 4 94% pdsfile/pds3file/rules/COUVIS_8xxx.py 71 0 24 0 100% pdsfile/pds3file/rules/COVIMS_0xxx.py 98 2 44 2 97% pdsfile/pds3file/rules/COVIMS_8xxx.py 69 0 26 0 100% pdsfile/pds3file/rules/EBROCC_xxxx.py 73 0 30 0 100% pdsfile/pds3file/rules/GO_0xxx.py 128 0 46 0 100% pdsfile/pds3file/rules/HSTxx_xxxx.py 80 0 30 0 100% pdsfile/pds3file/rules/JNOJIR_xxxx.py 23 0 0 0 100% pdsfile/pds3file/rules/JNOJNC_xxxx.py 17 0 0 0 100% pdsfile/pds3file/rules/JNOSP_xxxx.py 13 0 0 0 100% pdsfile/pds3file/rules/NHSP_xxxx.py 13 0 0 0 100% pdsfile/pds3file/rules/NHxxxx_xxxx.py 120 0 52 0 100% pdsfile/pds3file/rules/RES_xxxx.py 6 0 0 0 100% pdsfile/pds3file/rules/RPX_xxxx.py 29 3 2 0 84% pdsfile/pds3file/rules/VGIRIS_xxxx.py 10 0 0 0 100% pdsfile/pds3file/rules/VGISS_xxxx.py 71 0 24 0 100% pdsfile/pds3file/rules/VG_0xxx.py 26 0 0 0 100% pdsfile/pds3file/rules/VG_20xx.py 10 0 0 0 100% pdsfile/pds3file/rules/VG_28xx.py 55 0 0 0 100% pdsfile/pds3file/rules/init.py 25 0 0 0 100% pdsfile/pds3file/rules/pytest_support.py 19 2 0 0 89% pdsfile/pds3file/tests/init.py 0 0 0 0 100% pdsfile/pds3file/tests/helper.py 9 2 0 0 78% pdsfile/pds3file/tests/test_pds3file_blackbox.py 593 1 46 1 99% pdsfile/pds3file/tests/test_pds3file_blackbox_cached.py 275 0 18 0 100% pdsfile/pds3file/tests/test_pds3file_whitebox.py 301 1 34 1 99% pdsfile/pds3file/tests/test_pdsviewable_blackbox.py 24 0 0 0 100% pdsfile/pds4file/init.py 68 9 6 2 82% pdsfile/pds4file/rules/init.py 25 0 0 0 100% pdsfile/pds4file/rules/cassini_iss.py 75 29 26 0 46% pdsfile/pds4file/rules/cassini_vims.py 75 29 26 0 46% pdsfile/pds4file/rules/pytest_support.py 19 7 0 0 63% pdsfile/pds4file/rules/uranus_occs_earthbased.py 75 29 26 0 46% pdsfile/pds4file/tests/init.py 0 0 0 0 100% pdsfile/pds4file/tests/helper.py 5 0 0 0 100% pdsfile/pds4file/tests/test_pds4file_blackbox.py 84 1 4 1 98% pdsfile/pdsfile.py 3002 364 1485 176 86% pdsfile/pdsviewable.py 269 95 124 22 59% pdsfile/preload_and_cache.py 23 5 10 2 79%

TOTAL 6473 622 2299 216 88% (venv) yu-jenchang make_rms_pdsfile_standalone rms-pdsfile $

SETI / rms-pdsfile

Make `rms-pdsfile` standalone #20