Adds a try-except block for datasets that may occasionally fail

Yale-LILY / SummerTime

An open-source text summarization toolkit for non-experts. EMNLP'2021 Demo

https://arxiv.org/abs/2108.12738

Apache License 2.0

264 stars 30 forks source link

Adds a try-except block for datasets that may occasionally fail #57

Closed MuroriM closed 3 years ago

MuroriM commented 3 years ago

Creates a 'loading_dataset' function wrapper that has a try-except block to catch when the dataset trying to be loaded cannot be reached online.
Implemented for the MLsum Dataset, which occasionally has this issue

niansong1996 commented 3 years ago

@MuroriM Lmk when this is done. It seems to me that the passing of #60 actually depends on the changes made here

MuroriM commented 3 years ago

@niansong1996 I've added the delay between retries.

MuroriM commented 3 years ago

@MuroriM Sorry, accidentally closed the PR. I've finished with all the planned changes. Let me know if there's an issue

MuroriM commented 3 years ago

@niansong1996, thanks for the catch. I've fixed the typo

niansong1996 commented 3 years ago

There is a small conflict which I fixed myself. Should be able to merge after the CI check.

MuroriM commented 3 years ago

@niansong1996 There was a conflict with the imports between this branch and the notebook branch which has just been merged. I've removed the sys module since it's no longer needed and added the time and json modules to resolve the conflict. Let me know if this is an issue.

MuroriM commented 3 years ago

@niansong1996 sorry, I hadn't seen your message. I'll be waiting 10 min after your comments before making any changes to avoid this issue in future

niansong1996 commented 3 years ago

Ah, that's totally okay. Actually I shouldn't have make commits on this branch, my bad. Merging it now.