`esmvalcore.esgf.download` fails for localfiles

[Peter9192]

from esmvalcore.config import CFG
from esmvalcore.dataset import Dataset
from esmvalcore.esgf import download

CFG['offline'] = False
CFG['always_search_esgf'] = True

dataset = Dataset(
    short_name='tos',
    mip='Omon',
    project='CMIP6',
    exp=['historical', 'ssp585'],  # will be concatenated
    dataset='IPSL-CM6A-LR',
    institute='*',
    ensemble='*',
    grid='*',
)
dataset.add_ancillary(short_name='areacella', mip='fx', ensemble='r0i0p0')
dataset.augment_facets()
dataset.find_files()
download(dataset.files, CFG['download_dir'])

First attempt at this worked but halfway through the download my internet connection was interrupted. Executing again resulted in

AttributeError: 'LocalFile' object has no attribute 'dataset'

I can workaround it by excluding the localfiles, but that requires importing the type from a hidden module esmvalcore.esgf._download. It would be nicer if ESMValCore could skip these automatically.

[valeriupredoi]

cheers @Peter9192 :beer: Was your internet connection working when you restarted it? ie was the script trying to "download" local files in an equivalent mode to offline=True if the internet was not working? Also - it could be that when the internet stopped working a file that was half-downloaded still appeared as a valid local file, whereas it's just a shell and it's corrupted

[bouweandela]

LocalFile is part of the public API and can be imported from esmvalcore.local.LocalFile, so that should help with filtering. However, I do agree that it would be nicer if they were automatically skipped.

[bouweandela]

esmvalcore.esgf.LocalFile is actually a subclass of pathlib.Path, so you could also filter on that. Or filter on esmvalcore.esgf.ESGFFile, as that is the expected input data type for esmvalcore.esgf.download.