Different behaviour between engines in xarray.Dataset.to_netcdf

What happened: Depending on the engine used writing a xarray.Dataset the behaviour can change notably using engine='scipy' will overwrite the file even if another file is using it. I am not sure this is a pure `xarray problem but would nice to be consistent ....

What you expected to happen: All engines throw an error if the file is being used

Minimal Complete Verifiable Example: Open 1 terminal and run:

import fasteners

lock = fasteners.InterProcessLock('test.nc') 
lock.acquire()

This is just to create a locked file, you can do this in another manner.

This is the bug:

import xarray as xr

da = xr.DataArray(
    [1, 2, 3, 4],
    dims=['lat'],
    name='lat'
)
ds = da.to_dataset()

ds.to_netcdf('test.nc', engine='netcdf4')
-> PermissionError: [Errno 13] Permission denied: b'/XXXX/XXX/bug.nc'

ds.to_netcdf('.nc', engine='h5netcdf')
-> OSError: Unable to create file (unable to lock file, errno = 35, error message = 'Resource temporarily unavailable')

ds.to_netcdf('.nc', engine='scipy')
-> no output (Saves file with no problems??)

Anything else we need to know?:

Environment:

Output of xr.show_versions()

INSTALLED VERSIONS ------------------ commit: None python: 3.8.5 (default, Sep 4 2020, 02:22:02) [Clang 10.0.0 ] python-bits: 64 OS: Darwin OS-release: 20.2.0 machine: x86_64 processor: i386 byteorder: little LC_ALL: None LANG: None LOCALE: None.UTF-8 libhdf5: 1.12.0 libnetcdf: 4.7.4 xarray: 0.16.2 pandas: 1.2.0 numpy: 1.19.4 scipy: 1.5.4 netCDF4: 1.5.5.1 pydap: None h5netcdf: 0.8.1 h5py: 3.1.0 Nio: None zarr: None cftime: 1.3.0 nc_time_axis: None PseudoNetCDF: None rasterio: None cfgrib: None iris: None bottleneck: None dask: 2020.12.0 distributed: 2020.12.0 matplotlib: 3.3.3 cartopy: None seaborn: None numbagg: None pint: None setuptools: 51.0.0.post20201207 pip: 20.3.3 conda: None pytest: 6.2.1 IPython: 7.19.0 sphinx: None

pydata / xarray

Different behaviour between engines in xarray.Dataset.to_netcdf #4790