As a user, I don't want to run into errors when updating the application when offline

[christophertubbs]

This pertains to ./scripts/update_package.sh

My internet went down this morning and I needed to update my codebase so all my changes were in place so I could test.

With everything already available locally (including all remote sources), I ran into issues like:

WARNING: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<pip._vendor.urllib3.connection.HTTPSConnection object at 0x1068942e0>: Failed to establish a new connection: [Errno 8] nodename nor servname provided, or not known')': /simple/dmod-hydrofabric/
WARNING: Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<pip._vendor.urllib3.connection.HTTPSConnection object at 0x1068944c0>: Failed to establish a new connection: [Errno 8] nodename nor servname provided, or not known')': /simple/dmod-hydrofabric/
Processing ./python/lib/hydrofabric/dist/dmod_hydrofabric-0.0.1-py3-none-any.whl
WARNING: Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<pip._vendor.urllib3.connection.HTTPSConnection object at 0x106882f70>: Failed to establish a new connection: [Errno 8] nodename nor servname provided, or not known')': /simple/dmod-metrics/
WARNING: Retrying (Retry(total=3, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<pip._vendor.urllib3.connection.HTTPSConnection object at 0x106882d00>: Failed to establish a new connection: [Errno 8] nodename nor servname provided, or not known')': /simple/dmod-metrics/
WARNING: Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<pip._vendor.urllib3.connection.HTTPSConnection object at 0x106882a60>: Failed to establish a new connection: [Errno 8] nodename nor servname provided, or not known')': /simple/dmod-metrics/
WARNING: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<pip._vendor.urllib3.connection.HTTPSConnection object at 0x106882910>: Failed to establish a new connection: [Errno 8] nodename nor servname provided, or not known')': /simple/dmod-metrics/
WARNING: Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<pip._vendor.urllib3.connection.HTTPSConnection object at 0x106880b50>: Failed to establish a new connection: [Errno 8] nodename nor servname provided, or not known')': /simple/dmod-metrics/
Processing ./python/lib/metrics/dist/dmod_metrics-0.2.0-py3-none-any.whl
WARNING: Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<pip._vendor.urllib3.connection.HTTPSConnection object at 0x106894cd0>: Failed to establish a new connection: [Errno 8] nodename nor servname provided, or not known')': /simple/dmod-modeldata/
WARNING: Retrying (Retry(total=3, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<pip._vendor.urllib3.connection.HTTPSConnection object at 0x106894eb0>: Failed to establish a new connection: [Errno 8] nodename nor servname provided, or not known')': /simple/dmod-modeldata/
WARNING: Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<pip._vendor.urllib3.connection.HTTPSConnection object at 0x106894fa0>: Failed to establish a new connection: [Errno 8] nodename nor servname provided, or not known')': /simple/dmod-modeldata/
WARNING: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<pip._vendor.urllib3.connection.HTTPSConnection object at 0x1068a21f0>: Failed to establish a new connection: [Errno 8] nodename nor servname provided, or not known')': /simple/dmod-modeldata/
WARNING: Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<pip._vendor.urllib3.connection.HTTPSConnection object at 0x1068a23a0>: Failed to establish a new connection: [Errno 8] nodename nor servname provided, or not known')': /simple/dmod-modeldata/
Processing ./python/lib/modeldata/dist/dmod_modeldata-0.12.0-py3-none-any.whl

An internet connection should not be required 100% of the time just to develop.

Either handling needs to be added to avoid having to call pip whenever moving code into the virtual environment or update_script.sh needs to be updated to 'install' in edit mode where packages don't need to be in the virtual environment to begin with.

[robertbartel]

It may be worth discussing whether we should move to editable mode as a general expectation for deployments, development or otherwise. I'm not sure I like it, but it would make certain things simpler.

Assuming we don't expect that for every deployment, I prefer not to use editable mode with DMOD's current multi-package setup. I think it's better to require ourselves to have the packages we use installed the same way for development as they will be for deployments. I see it as worth the small extra overhead, though I'm open to alternative ideas if there's a more convenient way to ensure we have this kind of environmental consistency for testing.

Regardless, I agree we should support at least some development activity without an active Internet connection. Probably the simplest solution here is to add an option to _scripts/updatepackage.sh that causes pip's --no-deps flag to be used.

This may be another sign that we need to (eventually) not have a bunch of individual packages like this all within the same repository, so we can potentially have that conversation as well (though perhaps in a different issue).

[aaraney]

Im fairly certain this is unavoidable because of how we specify / need to specify several dependencies. To my best knowledge, pip will re-pull any dependencies that are specified using a url like syntax (e.g. git+https://github.com/noaa-owp/hypy@master#egg=hypy&subdirectory=python) b.c. pip cannot be certain of the version. Like, @robertbartel mentioned, I think the best you can do is use pip's --no-deps flag in this case. If noaa-owp python repos were uploaded to pypi this would not be an issue.