pattern-matching getDirectoryDirs

[alexanderkjeldaas]

I have the following directory structure:

data/{name-of-node}/{date}/data

I want to 'need' data///some-file. Ideally I'd want to write something like:

action $ do
    existing <- getDirectoryDirs "data/*/*"
    need $ map (++ "/some-file") existing

But my toolbox only includes getDirectoryFiles which can't return the required directories. For my use-case I don't need a specific getDirectoryDirs - a function that does pattern-matching on all entries in a directory would be fine.

This issue also shows a pretty comon pattern for me - using getDirectoryFiles to derive a set of files I 'want', but I have to use action + need since I can't use getDiretoryFiles in the Rules monad.

[ndmitchell]

You should be able to write:

action $ do
      d1 <- getDirectoryDirs "data"
      d2 <- fmap concat $ mapM getDirectoryDirs d1
      need $ map (</> "some-file") d2

getDirectoryDirs isn't perfect for you here, since it takes a FilePath not a FilePattern. I wonder if getDirectoryDirs should be enhanced to look like getDirectoryFiles, or if it's not common enough to be needed?

The definition of want is simply action . need so having to do action/need is absolutely fine - in fact it's entirely deliberate since we can guarantee that the untracked information from getDirectoryDirs is not used in any other rules, making it harder to miss dependencies.

[alexanderkjeldaas]

Agreed, it's probably a good idea to keep getDirectoryDirs ouside of Rules.

However getDirectoryDirs returns relative paths. I wrote something like this:

  n <- return . map ("data/" ++) =<< ( getDirectoryDirs "data/")
  forM_ n $ \x -> need . map  (\y -> x ++ "/" ++ y ++ "/somefile") =<<

getDirectoryDirs x

Which is not exactly beautiful :-)

On Sat, Jul 6, 2013 at 12:33 AM, Neil Mitchell notifications@github.comwrote:

You should be able to write:

action $ do d1 <- getDirectoryDirs "data" d2 <- fmap concat $ mapM getDirectoryDirs d1 need $ map (</> "some-file") d2

getDirectoryDirs isn't perfect for you here, since it takes a FilePathnot a FilePattern. I wonder if getDirectoryDirs should be enhanced to look like getDirectoryFiles, or if it's not common enough to be needed?

The definition of want is simply action . need so having to do action/needis absolutely fine - in fact it's entirely deliberate since we can guarantee that the untracked information from getDirectoryDirs is not used in any other rules, making it harder to miss dependencies.

— Reply to this email directly or view it on GitHubhttps://github.com/ndmitchell/shake/issues/29#issuecomment-20542060 .

[ndmitchell]

I now remember why I didn't make getDirectoryDirs use patterns. Typically you want to do things like delete ".git" on the list of directories, so it's better to hand roll it as the patterns don't really work.

I think if you define getDirectoryDirsFull which does the get and prepends the filepath (so makes them not relative) as a single operation, then it looks passable. I wonder if getDirectoryDirs or getDirectoryFiles in Shake should always have included the path in the result, but I suspect it is too late to fix now.

[ndmitchell]

Hmm, I think there is some scope for a new function. One option is to introduce getDirectoryDirectories (ugly name). Another option is to make getDirectoryFiles treat patterns which end with a / (but not //) as searching for directories (but the function is called Files and returns directories, which would be weird). See also #108, which probably wants tackling at the same time.