Remote access to dataset. E.g. loading copernicus data remot

[thogar-computer]

Remote access to dataset. E.g. loading Copernicus data remotely we have 3 examples of data access

BODC is to look over the examples and review them to see which would fit within COAST, given coding styles and long term stability

added question xarray lazy loads data - how does that work over streaming data

AC a report (wiki) on what option might be the best, share any scripts with team.

[thogar-computer]

a possible example of this by tom prime - might be useful to look at these

[thogar-computer]

@MCazaly to add link to python lib that might do this work

[thogar-computer]

@katavouta share example of code

[katavouta]

Sent example by email to @thogar-computer

My example is for downloading a subset of data from copernicus using MOTU client. (I do not have an example or know how to load copernicus data remotely).

[thogar-computer]

@anwiseNOCL and @thogar-computer to test from a Mac and windows view point

[MCazaly]

An account here is needed for the test: https://marine.copernicus.eu/

This is the module with the test I want non-Linux people to run: https://github.com/British-Oceanographic-Data-Centre/COAsT/blob/555f520eb3eba64a9c11ffa5503d7cc05f6629c5/tests/test_copernicus_integration.py

[thogar-computer]

@MCazaly my tests seem to have passed - results included below

(COAsT-555f520eb3eba64a9c11ffa5503d7cc05f6629c5) C:\work\code\COAsT-555f520eb3eba64a9c11ffa5503d7cc05f6629c5\COAsT-555f520eb3eba64a9c11ffa5503d7cc05f6629c5>pytest tests\test_copernicus_integration.py 
========================================================================================== test session starts ==========================================================================================
platform win32 -- Python 3.8.13, pytest-7.1.1, pluggy-1.0.0
rootdir: C:\work\code\COAsT-555f520eb3eba64a9c11ffa5503d7cc05f6629c5\COAsT-555f520eb3eba64a9c11ffa5503d7cc05f6629c5
plugins: mock-3.7.0
collected 1 item

tests\test_copernicus_integration.py .                                                                                                                                                             [100%]

=========================================================================================== warnings summary ============================================================================================ 
..\..\..\Miniconda3\envs\COAsT-555f520eb3eba64a9c11ffa5503d7cc05f6629c5\lib\site-packages\pydap\model.py:175
  C:\work\Miniconda3\envs\COAsT-555f520eb3eba64a9c11ffa5503d7cc05f6629c5\lib\site-packages\pydap\model.py:175: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collect
ions.abc' is deprecated since Python 3.3, and in 3.10 it will stop working
    from collections import OrderedDict, Mapping

..\..\..\Miniconda3\envs\COAsT-555f520eb3eba64a9c11ffa5503d7cc05f6629c5\lib\site-packages\pydap\responses\das.py:14
  C:\work\Miniconda3\envs\COAsT-555f520eb3eba64a9c11ffa5503d7cc05f6629c5\lib\site-packages\pydap\responses\das.py:14: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from '
collections.abc' is deprecated since Python 3.3, and in 3.10 it will stop working
    from collections import Iterable

tests/test_copernicus_integration.py::test_get_product
tests/test_copernicus_integration.py::test_get_product
tests/test_copernicus_integration.py::test_get_product
tests/test_copernicus_integration.py::test_get_product
tests/test_copernicus_integration.py::test_get_product
tests/test_copernicus_integration.py::test_get_product
  C:\work\Miniconda3\envs\COAsT-555f520eb3eba64a9c11ffa5503d7cc05f6629c5\lib\site-packages\pydap\handlers\dap.py:366: DeprecationWarning: The binary mode of fromstring is deprecated, as it behaves surpr
isingly on unicode inputs. Use frombuffer instead
    n = np.fromstring(stream.read(4), DAP2_ARRAY_LENGTH_NUMPY_TYPE)[0]

tests/test_copernicus_integration.py::test_get_product
tests/test_copernicus_integration.py::test_get_product
tests/test_copernicus_integration.py::test_get_product
tests/test_copernicus_integration.py::test_get_product
tests/test_copernicus_integration.py::test_get_product
tests/test_copernicus_integration.py::test_get_product
  C:\work\Miniconda3\envs\COAsT-555f520eb3eba64a9c11ffa5503d7cc05f6629c5\lib\site-packages\pydap\handlers\dap.py:383: DeprecationWarning: The binary mode of fromstring is deprecated, as it behaves surpr
isingly on unicode inputs. Use frombuffer instead
    np.fromstring(

tests/test_copernicus_integration.py::test_get_product
  C:\work\code\COAsT-555f520eb3eba64a9c11ffa5503d7cc05f6629c5\COAsT-555f520eb3eba64a9c11ffa5503d7cc05f6629c5\coast\_utils\logging_util.py:78: UserWarning: C:\work\code\COAsT-555f520eb3eba64a9c11ffa5503d
7cc05f6629c5\COAsT-555f520eb3eba64a9c11ffa5503d7cc05f6629c5\coast\data\gridded.py._setup_grid_obj.79: No NEMO domain specified, only limited functionality will be available
    return warnings.warn(add_info(msg), *args, **kwargs)

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html

[thogar-computer]

do we need to look into the two 3.10 warnings above?

[MCazaly]

They're from Pydap so largely out of our control... It means COAsT won't be able to go to 3.10 until that's resolved.

[thogar-computer]

@thogar-computer @MCazaly look into best way to set max version of python to top getting to 3.10 by mistake

[MCazaly]

@thogar-computer @MCazaly look into best way to set max version of python to top getting to 3.10 by mistake

See this commit.

[MCazaly]

This has now been merged into develop.