Inconsistent operations on paths containing characters outside of range

marctrem commented 7 years ago

Holla,

Using hadoop-manta through Kafka Connect HDFS discloses a strange behavior.

Paths that contain characters that are outside of the ones allowed by Manta are created without warning. The out-of-range characters get escaped.

Ex.: ~~/stor/project/topics/+tmp/topic_name/partition=0 -> ~~/stor/project/topics/%2Btmp/topic_name/partition%3D0 (which is okay, I guess)

The problem arises when kafka-connect-hdfs tries to read back from there: Manta returns a 404.

[1:statusLine=HTTP/1.1 404 Not Found]
[2:serverCode=DirectoryDoesNotExist]
[3:server_message=/<account>/stor/project/topics/+tmp/topic_name/partition=0 does not exist]

Looks like the conversion happens silently on the Manta side, on creation. (I might be wrong I did not dig yet.)

Should:

hadoop-manta escape the path somewhere like here before even involving Manta
Manta be responsible and either: prevent creation of such items OR convert consistently

Please let me know if this is a real issue or/and if I am getting something wrong.

Regards, Marc

dekobon commented 7 years ago

My gut instinct is that this issue is due to this problem with the Manta SDK: https://github.com/joyent/java-manta/pull/234

It should be fixed in the upcoming release of the Manta SDK.

dekobon commented 7 years ago

Fixed in 54b5cdfd21de1fc4ad7d51431621123d97622f5b

TritonDataCenter / hadoop-manta

Inconsistent operations on paths containing characters outside of range #5