[Bug] `summarize_results_all_tasks_all_devs` only returns results for first seed found.

[eddiebergman]

Not really sure how the whole task and dev id's really work but from what I gather from this function the path goes something like this:

• If sub_dir: workingdir/seed/sub_dir/task_id/dev_id.
• Else: workingdir/seed/task_id/dev_id

However this function only takes it over the first valid seed dir found, with no knowledge of what this is.

https://github.com/automl/neps/blob/dbc307697168a4b2099bc117e200e42ab36c2fd9/neps/utils/data_loading.py#L223-L251

---

I guess the way to go (if seed dir is still relevant) is to specifically ask for what seed is requested in the function params.

[eddiebergman]

As an additional thing to fix, this function calls read_tasks_and_dev_stages_from_disk which essentially reads all results from disk, uses just the ids it retrieves and the proceeds to call anther function summarize_results which will then load all results again.

[eddiebergman]

Last point, nothing in NePS is using this function and there's no publicised user API for this. I guess we'll need to rediscover the point of this feature since it's pretty undocumented anywhere.

My guess it's some sort of convinience feature but I don't feel this provides much benefit and perhaps more confusion than if someone just made a different working_directory

[eddiebergman]

More issues with this function, pretty sure this would just not work if subdir was given.

https://github.com/automl/neps/blob/c48417ad561da443e57c07b6040d13f44a61e728/neps/utils/data_loading.py#L242

[eddiebergman]

What even is a user prior dir... what's meant to be in there that's different from any other run directory?

https://github.com/automl/neps/blob/c48417ad561da443e57c07b6040d13f44a61e728/neps/utils/data_loading.py#L240