Omit old and non-relevant datasets from early alert tool

[timadriaens]

Hi, as this represents an early alert tool, imo, some datasets on gbif with occurrences for Belgium are actually not that relevant (I have not checked them all):

• [ ] The Flora checklist for instance, has data only until 1971 which is not relevant at all for early warning
• [ ] maybe we should be focusing on human obervations?

[damianooldoni]

1. It's just a question of tradeoff indeed. We could trigger a download starting from 2000 for example. This will reduce the size of the data when we will include more species.l in case it would be a critical problem (see #213 )

2. As we are going to broaden up the species list, we should not exclude other kind of observations as machine observations.

I am in favour to improve the download query.

[timadriaens]

yes, I think that is a good idea, it will automatically make the list of datasets shorter and more relevant (focus more on 'living datasets').

[niconoe]

Implemented on dev, let's test this!

[damianooldoni]

Thanks @niconoe. It seems we filtered out ~2.8k observations. On dev alert tool:

Data import: #448 -> 57.896 observations Data import: #449 -> 55.118 observations

[damianooldoni]

About involved datasets: Data import: #448 -> 62 datasets Data import: #449 -> 47 datasets

So, @timadriaens, I think this filtering is quite effective as 15 datasets are removed from the alert tool.

[niconoe]

Yep, please note that for now those datasets still appear in the filters because of #222. But the more I think of it, the more I think we can be a bit more aggressive and actually remove those empty datasets for the system.

[timadriaens]

Yes, noticed that on the devtools, I think it would be best to remove them (as also people would not be able to select occurrences on those).

[damianooldoni]

Yes, @niconoe, I agree on it. Being a little aggressive is good sometimes 🤣

[damianooldoni]

@niconoe: Is this done? I mean, did you implement https://github.com/riparias/gbif-alert/issues/215#issuecomment-1525598760? We can close this then.

[niconoe]

@damianooldoni: I just implemented #222, so we can now close this!