search

toddfarmer / arrow-migration

0 stars 1 forks source link

[Python] Document that HadoopFileSystem.mkdir with create_parents=False has no effect #1402

Closed toddfarmer closed 7 years ago

toddfarmer commented 7 years ago

Note: This issue was originally created as ARROW-1458. Please see the migration documentation for further details.

Original Issue Description:

Even when setting the create_parents argument to False, mkdir still creates all parent directories.

toddfarmer commented 7 years ago

Note: Comment by Wes McKinney (wesm): I think this is a "feature" of HDFS, hdfsCreateDirectory always will create parents. There is some benefit to all filesystems having the same API, though

toddfarmer commented 7 years ago

Note: Comment by Brecht Machiels (brechtm): Oh, I didn't know that. The hadoop fs -mkdir command does take a -p option though and the default is to not create parent dirs.

toddfarmer commented 7 years ago

Note: Comment by Wes McKinney (wesm): I see, it must just be a limitation of libhdfs, then

toddfarmer commented 7 years ago

Note: Comment by Wes McKinney (wesm): I marked this for 0.7.0 and changed title to make this a documentation thing

toddfarmer commented 7 years ago

Note: Comment by Wes McKinney (wesm): PR: https://github.com/apache/arrow/pull/1041

toddfarmer commented 7 years ago

Note: Comment by Wes McKinney (wesm): Issue resolved by pull request 1041 https://github.com/apache/arrow/pull/1041