search

NVIDIA / spark-rapids-tools

User tools for Spark RAPIDS
Apache License 2.0
49 stars 35 forks source link

[FEA] Enable recursive lookup for event log paths by default #1177

Closed parthosa closed 2 weeks ago

parthosa commented 1 month ago

Currently, we support recursive searching of event log paths using wildcards. We should enable recursive lookup by default for better usability.

Example, If the event logs path is /path/eventlog/*/*, Q/P tool would recursively search for event logs in that path but if the event logs path is /path/eventlog Q/P tool would not search recursively.

Going forward, we want to recursively search for event logs even when /path/eventlog is passed as input.

tgravescs commented 1 month ago

what tool is this for? I thought this was added for certain ones, maybe this never got integrated, I know we had an issue for it. My old branch is here: https://github.com/tgravescs/spark-rapids-tools/tree/recurseDirsEventlogs

parthosa commented 1 month ago

The request is for Qual tool. But we can add for Profile tool as well. Updating the PR description with an example.

amahussein commented 4 weeks ago

@parthosa there was an old issue https://github.com/NVIDIA/spark-rapids-tools/issues/167 opened for this. Please close one of them as duplicate if it has the same requirement

parthosa commented 4 weeks ago

Thanks @amahussein. Closing the previous one since this has a roadmap assigned.