openclimatefix / ocf_datapipes

OCF's DataPipe based dataloader for training and inference
MIT License
13 stars 11 forks source link

Reduce repetition in data source config validation #279

Closed Sukh-P closed 6 months ago

Sukh-P commented 7 months ago

Detailed Description

At the moment in the open_and_return_data_pipes function there is some repetition of code since there are repeated checks on each data source which checks a bool, an input configuration is not none and filepath/filename is not an empty string, see here for an example

Possible Implementation

Write a helper function which does these checks given some inputs to reduce repeated code

aung-st commented 7 months ago

Hi, can I take this issue? I'm thinking about making a nested helper function to reduce the repetition.

roshnaeem commented 7 months ago

Hello @Sukh-P, i would like to work on this issue. If it is available can i start working on it?

Sukh-P commented 6 months ago

Hi @aung-st and @roshnaeem, @aung-st since you had commented on here first yes please go ahead, that sounds like it would be useful, if you have more questions or updates you can add them here as comments and tag me on any PRs and I'll have a look!

@roshnaeem we will be creating more good first issues soon so hopefully you will have others to work on, hope that's alright.

Thanks both

Sukh-P commented 6 months ago

@aung-st, just checking in are you working on this/still planning on working this? No worries if not, can reassign if that's the case, thanks!

aung-st commented 6 months ago

@Sukh-P, yeah sorry I don't think I have the time to work on this right now so best to reassign to make way for someone who can. Thanks.

Sukh-P commented 6 months ago

@roshnaeem let me know if you still would like to work on this one , thanks

roshnaeem commented 6 months ago

Sure @Sukh-P.

roshnaeem commented 6 months ago

Hello @Sukh-P, I opened PR #291, please check and let me know if any changes are required.