conda recipe for pysolid

[yunjunz]

Open an issue here to keep track of things for pysolid, so that https://github.com/insarlab/MintPy/issues/648 could be focused on mintpy itself.

@pbrotoisworo are you able to use pysolid on Windows? If so, how do you install it, especially how do you compile the Fortran code?

[pbrotoisworo]

@yunjunz I had no idea PySolid was not supported on Windows and I've been able to run analyses all this time and there were no errors regarding PySolid. But I think it's because it doesn't run the SET analysis. I assume it only runs correct_SET in certain situations only? I just checked my logs and here is the relevant section:

******************** step - correct_SET ********************
Input data seems to be geocoded. Lookup file not needed.
No solid Earth tides correction.

PySolid 0.1.2 is installed and shows up when I type conda list or pip list. Here is an image of the pysolid folder in site-packages:

I assume that's the uncompiled Fortran code?

If you are curious. Here is the YML file I used to prepare my MintPy environment in a Windows environment. It's based on the MintPy environment YML file but I've modified it to include some extra libraries for hyp3, PyQt, sentinelsat, etc.

As of writing, I am able to just type the conda env create command and then I can already start doing MintPy analyses.

# conda env create -f <FILE>.yml
name: mintpy
channels:
  - conda-forge
  - defaults
  - anaconda
dependencies:
  - python>=3.6,<3.9
  - git
  - cartopy
  - cvxopt
  - dask>=1.0
  - dask-jobqueue>=0.3
  - defusedxml
  - fiona
  - h5py<3
  - hyp3_sdk
  - lxml
  - matplotlib
  - numpy
  - pip
  - pyproj
  - scikit-image
  - scipy
  - pip:
    - git+https://github.com/insarlab/PySolid.git
    - git+https://github.com/tylere/pykml.git
    - git+https://github.com/yunjunz/PyAPS.git
    - git+https://github.com/insarlab/MintPy.git
    - git+https://github.com/ASFHyP3/hyp3-lib.git
    - sentinelsat
    - PyQt5
    - PyQt5Designer
    - pytest
  # pyaps dependencies
  - cdsapi
  - ecCodes
  - netcdf4
  - pygrib
  - pyhdf
  # pyresample dependencies
  - pyresample
  - openmp
  - pykdtree
  - xarray
  - zarr

[yunjunz]

Thank you @pbrotoisworo for the info. It looks like the pre-compiled version of pysolid is installed, but the pre-compiled repo does not include windows version.

Since the SET correction is turned off by default, so you have not used it yet, that's why no error was reported. SET only matters for very large-scale analysis, e.g. cross the entire Tibet, so it's totally fine to not use it in small to middle-scale analysis.

@pbrotoisworo, @jhkennedy and @Jlrine2 It seems like conda install -c conda-forge fortran-compiler supports windows based on https://anaconda.org/conda-forge/fortran-compiler/. It would be worth trying the installation note of compile from source on windows. No rush, as we could wait for recipes merged to conda-forge first, as @jhkennedy suggested.

[pbrotoisworo]

@yunjunz I can experiment with Fortran compilers on my Windows machine. I'll update if I am able to run the PySolid test file

[yunjunz]

That will be great @pbrotoisworo.

[pbrotoisworo]

@yunjunz Just tried it on my machine and it seems that it works but there's an issue with the test.

I've tried workarounds such as context managers and time.sleep() but it doesn't work. I think the Fortran code writes solid.txt but it does not release the process properly? I'm not exactly sure, I have no experience with Fortran. I'm still trying to fix this error.

Here is the log of the compiling results: https://pastebin.com/rxFrhFvn

Another issue is requirements.txt has fortran-compiler as a requirement but it cannot be found when I tried to install it through pip but I can install it when installing through conda install -c conda-forge fortran-compiler.

[yunjunz]

Thank you @pbrotoisworo. Your compilation looks good to me, which is good news.

I think the Fortran code writes solid.txt but it does not release the process properly?

I think so too. Could you add the following line to solid.for#L168 and solid.for#L310 and try again?

      close(lout)

fortran-compiler is only available through conda, that's also why I did not put it in setup.py for pip. Plus I am not aware that pip could run compilation also.

If the above change works, it would confirm that the conda installation recipe should work on windows as well.

[pbrotoisworo]

I updated to the latest commit (https://github.com/insarlab/PySolid/commit/65b1b7debf329e0bed4c0dd33791c0e14684a5e1) and recompiled PySolid with your suggestions.

test_SET_point.py is successful and I can view the figure. But I had to modify the code to open the figure. In Windows you can just input the path of the image from the terminal to open the image.

Here is the quick solution I did:

if sys.platform == 'win32':
    os.system(out_fig)
else:
    cmd = 'open'
    if sys.platform in ['linux']:
        cmd = 'display'
    os.system('{} {}'.format(cmd, out_fig))

test_SET_grid.py is not successful. The text file from Fortran creates empty values. Here is a sample of the first few lines (It just keeps repeating with 4 commas):

# program solid -- UTC version -- 2018jun01
# year, month, day = 2020 12 25
# hour, minute, second = 14  7 44
# S, N, W, E =   33.479166795   33.800000000 -118.200000000 -117.796666828
# step_lat, num_lat =   -0.009166663    35
# step_lon, num_lon =    0.009166663    44
,  ,  ,  ,  
,  ,  ,  ,  
,  ,  ,  ,  
,  ,  ,  , 
etc...

So it raises the error:

Traceback (most recent call last):
  File "test_SET_grid.py", line 33, in <module>
    tide_u) = pysolid.calc_solid_earth_tides_grid(dt_obj, atr, verbose=True)
  File "C:\Users\Angelo\Documents\PANJI\Projects\PySolid\pysolid\grid.py", line 95, in calc_solid_earth_tides_grid
    fc = np.loadtxt(txt_file,
  File "C:\Users\Angelo\Miniconda3\envs\pysolid\lib\site-packages\numpy\lib\npyio.py", line 1148, in loadtxt
    for x in read_data(_loadtxt_chunksize):
  File "C:\Users\Angelo\Miniconda3\envs\pysolid\lib\site-packages\numpy\lib\npyio.py", line 999, in read_data
    items = [conv(val) for (conv, val) in zip(converters, vals)]
  File "C:\Users\Angelo\Miniconda3\envs\pysolid\lib\site-packages\numpy\lib\npyio.py", line 999, in <listcomp>
    items = [conv(val) for (conv, val) in zip(converters, vals)]
  File "C:\Users\Angelo\Miniconda3\envs\pysolid\lib\site-packages\numpy\lib\npyio.py", line 736, in floatconv
    return float(x)
ValueError: could not convert string to float: ''

[yunjunz]

Thank you @pbrotoisworo, it's good to hear test_SET_point.py works. Could you issue a PR for the test_SET_point/grid.py modification?

I will update the solid.for code together with its pre-compiled *.so files in another PR.

Hmm..., the two functions in Fortran level are almost the same, so it's strange why only one of them works in Windows. Could you please open an issue for it so that we could keep track of it and hopefully someone might fix it?

[pbrotoisworo]

@yunjunz I've created PR https://github.com/insarlab/PySolid/pull/14. I've modified test_SET_point.py and test_SET_grid.py since the issue is present in both files.