Initially, I downloaded data from entire fields (psychology and economics) per year (27 years x 2 fields = 54 files) as I felt like was tidier to deal with than hundreds (27 years x 55 journals = 1485 files).
That said, there are benefits to storing data by journal instead of field (like it's currently done for the field "general"). First, it's easier to add new journals because then only the new journal data needs to be redownloaded, instead of the entire field. Second is that this helps prevent future situation where the workflow would fail because the field files would get too big to process.
For these reasons, I think in the future we should consider saving all files per year and journal even if that results in over a thousand files.
Initially, I downloaded data from entire fields (psychology and economics) per year (27 years x 2 fields = 54 files) as I felt like was tidier to deal with than hundreds (27 years x 55 journals = 1485 files).
That said, there are benefits to storing data by journal instead of field (like it's currently done for the field "general"). First, it's easier to add new journals because then only the new journal data needs to be redownloaded, instead of the entire field. Second is that this helps prevent future situation where the workflow would fail because the field files would get too big to process.
For these reasons, I think in the future we should consider saving all files per year and journal even if that results in over a thousand files.