cypher_authentication.rb fails to parse Neo4j response.

[tritonresearch]

This is for:

• neo4j (4.0.0.rc.1 b55bf8d)
• neo4j-core (3.1.0)
• Neo4j version 2.2.0

I am connecting to the Neo4j server when I run rails c, and get the following stack:

/Library/Ruby/Gems/2.0.0/gems/json-1.8.1/lib/json/common.rb:155:in `initialize': no implicit conversion of Hash into String (TypeError)
from /Library/Ruby/Gems/2.0.0/gems/json-1.8.1/lib/json/common.rb:155:in `new'
from /Library/Ruby/Gems/2.0.0/gems/json-1.8.1/lib/json/common.rb:155:in `parse'
from /Library/Ruby/Gems/2.0.0/gems/neo4j-core-3.1.0/lib/neo4j-server/cypher_authentication.rb:41:in `authenticate'
from /Library/Ruby/Gems/2.0.0/gems/neo4j-core-3.1.0/lib/neo4j-server/cypher_session.rb:51:in `open'
from /Library/Ruby/Gems/2.0.0/gems/neo4j-core-3.1.0/lib/neo4j-server/cypher_session.rb:5:in `block in <module:Server>'
from /Library/Ruby/Gems/2.0.0/gems/neo4j-core-3.1.0/lib/neo4j/session.rb:112:in `call'
from /Library/Ruby/Gems/2.0.0/gems/neo4j-core-3.1.0/lib/neo4j/session.rb:112:in `create_session'
from /Library/Ruby/Gems/2.0.0/gems/neo4j-core-3.1.0/lib/neo4j/session.rb:99:in `open'
from /Users/jbrukh/.bundler/ruby/2.0.0/neo4j-b55bf8d22fce/lib/neo4j/railtie.rb:57:in `open_neo4j_session'
from /Users/jbrukh/.bundler/ruby/2.0.0/neo4j-b55bf8d22fce/lib/neo4j/railtie.rb:73:in `block (2 levels) in <class:Railtie>'
from /Users/jbrukh/.bundler/ruby/2.0.0/neo4j-b55bf8d22fce/lib/neo4j/railtie.rb:72:in `each'
from /Users/jbrukh/.bundler/ruby/2.0.0/neo4j-b55bf8d22fce/lib/neo4j/railtie.rb:72:in `block in <class:Railtie>'
from /Library/Ruby/Gems/2.0.0/gems/railties-4.1.6/lib/rails/initializable.rb:30:in `instance_exec'
from /Library/Ruby/Gems/2.0.0/gems/railties-4.1.6/lib/rails/initializable.rb:30:in `run'
from /Library/Ruby/Gems/2.0.0/gems/railties-4.1.6/lib/rails/initializable.rb:55:in `block in run_initializers'
from /System/Library/Frameworks/Ruby.framework/Versions/2.0/usr/lib/ruby/2.0.0/tsort.rb:150:in `block in tsort_each'
from /System/Library/Frameworks/Ruby.framework/Versions/2.0/usr/lib/ruby/2.0.0/tsort.rb:183:in `block (2 levels) in each_strongly_connected_component'
from /System/Library/Frameworks/Ruby.framework/Versions/2.0/usr/lib/ruby/2.0.0/tsort.rb:219:in `each_strongly_connected_component_from'
from /System/Library/Frameworks/Ruby.framework/Versions/2.0/usr/lib/ruby/2.0.0/tsort.rb:182:in `block in each_strongly_connected_component'
from /System/Library/Frameworks/Ruby.framework/Versions/2.0/usr/lib/ruby/2.0.0/tsort.rb:180:in `each'
from /System/Library/Frameworks/Ruby.framework/Versions/2.0/usr/lib/ruby/2.0.0/tsort.rb:180:in `each_strongly_connected_component'
from /System/Library/Frameworks/Ruby.framework/Versions/2.0/usr/lib/ruby/2.0.0/tsort.rb:148:in `tsort_each'
from /Library/Ruby/Gems/2.0.0/gems/railties-4.1.6/lib/rails/initializable.rb:54:in `run_initializers'
from /Library/Ruby/Gems/2.0.0/gems/railties-4.1.6/lib/rails/application.rb:300:in `initialize!'
from /Users/jbrukh/Triton/rails/triton-app/config/environment.rb:5:in `<top (required)>'
from /Library/Ruby/Gems/2.0.0/gems/activesupport-4.1.6/lib/active_support/dependencies.rb:247:in `require'
from /Library/Ruby/Gems/2.0.0/gems/activesupport-4.1.6/lib/active_support/dependencies.rb:247:in `block in require'
from /Library/Ruby/Gems/2.0.0/gems/activesupport-4.1.6/lib/active_support/dependencies.rb:232:in `load_dependency'
from /Library/Ruby/Gems/2.0.0/gems/activesupport-4.1.6/lib/active_support/dependencies.rb:247:in `require'
from /Library/Ruby/Gems/2.0.0/gems/spring-1.1.3/lib/spring/application.rb:92:in `preload'
from /Library/Ruby/Gems/2.0.0/gems/spring-1.1.3/lib/spring/application.rb:140:in `serve'
from /Library/Ruby/Gems/2.0.0/gems/spring-1.1.3/lib/spring/application.rb:128:in `block in run'
from /Library/Ruby/Gems/2.0.0/gems/spring-1.1.3/lib/spring/application.rb:122:in `loop'
from /Library/Ruby/Gems/2.0.0/gems/spring-1.1.3/lib/spring/application.rb:122:in `run'
from /Library/Ruby/Gems/2.0.0/gems/spring-1.1.3/lib/spring/application/boot.rb:18:in `<top (required)>'
from /Library/Ruby/Site/2.0.0/rubygems/core_ext/kernel_require.rb:54:in `require'
from /Library/Ruby/Site/2.0.0/rubygems/core_ext/kernel_require.rb:54:in `require'
from -e:1:in `<main>'

The contents of the response body that it is failing on is:

{"username"=>"neo4j", "password_change"=>"http://myhost:7474/user/neo4j/password", "password_change_required"=>false, "authorization_token"=>"b997f89f681c5ac22f1438e557240217", "authorization_token_change"=>"http://myhost:7474/user/neo4j/authorization_token"}

Any ideas?

Thanks, Jake

[subvertallchris]

That's odd, it's getting a valid authentication response. You're running Neo4j 2.2.0, right?

[tritonresearch]

Yes, 2.2.0. The error seems to suggest that JSON.parse() is getting something other than serialized JSON. I'll check the type of data coming back.

[subvertallchris]

What does your server connection string look like?

[tritonresearch]

Apologies and I can't explain this, but after running bundle that error doesn't happen anymore. Closing this.

[subvertallchris]

This is always my favorite kind of problem.

[tritonresearch]

Ok, I have reproduced this. (Thanks for looking at this!)

So, I have a Neo4j 2.2.0 instance running on a server with HTTP Basic Auth. I can log into the admin console just fine. I can also access the database through other drivers using the same authentication scheme.

I then set up my development.rb as follows:

config.neo4j.session_type = :server_db
config.neo4j.session_path = ENV["MY_NEO4J_URL"]  # e.g. http://myhost:7474

After this, running rails c results in the expected stack:

cypher_authentication.rb:61:in `rescue in obtain_token': Neo4j authentication is enabled, username/password are required but missing (Neo4j::Server::CypherAuthentication::MissingCredentialsError)

I then add config:

config.neo4j.session_options = { basic_auth: { username: 'jake', password: 'abc123'}  }

Which results in the original stack I posted above. The JSON.parse chokes on the response.body because it is of type Hash. (e.g. same error as if your run JSON.parse({:a => 1})).

How can I fish out the server connection string?

[subvertallchris]

I was looking for your session options and path, so we're good there. Neo4j authentication is enabled, username/password are required but missing (Neo4j::Server::CypherAuthentication::MissingCredentialsError) is defined here but I'm not sure why you're getting it. Give me a sec to test something on my machine.

EDIT, my bad, read that too fast, I see what you mean now.

[tritonresearch]

I was originally getting it because I didn't include the credentials, so that is expected. Once the credentials are added, though, we get the Hash response which doesn't parse in JSON.

[subvertallchris]

Going to be a couple minutes before I can mess with it more. In the meantime, can you tell me what happens if you include oj and oj_mimic_json in your gemfile? I just realized that's how I was playing with it in my test app.

[tritonresearch]

Trying this. One moment.

[tritonresearch]

It results in a similar error, with the following stack:

/Library/Ruby/Gems/2.0.0/gems/neo4j-core-3.1.0/lib/neo4j-server/cypher_authentication.rb:40:in `parse': strict_parse() expected a String or IO Object. (ArgumentError)
from /Library/Ruby/Gems/2.0.0/gems/neo4j-core-3.1.0/lib/neo4j-server/cypher_authentication.rb:40:in `authenticate'
from /Library/Ruby/Gems/2.0.0/gems/neo4j-core-3.1.0/lib/neo4j-server/cypher_session.rb:51:in `open'
from /Library/Ruby/Gems/2.0.0/gems/neo4j-core-3.1.0/lib/neo4j-server/cypher_session.rb:5:in `block in <module:Server>'
from /Library/Ruby/Gems/2.0.0/gems/neo4j-core-3.1.0/lib/neo4j/session.rb:112:in `call'
from /Library/Ruby/Gems/2.0.0/gems/neo4j-core-3.1.0/lib/neo4j/session.rb:112:in `create_session'
from /Library/Ruby/Gems/2.0.0/gems/neo4j-core-3.1.0/lib/neo4j/session.rb:99:in `open'
from /Users/jbrukh/.bundler/ruby/2.0.0/neo4j-a05bcc05c6f0/lib/neo4j/railtie.rb:57:in `open_neo4j_session'
from /Users/jbrukh/.bundler/ruby/2.0.0/neo4j-a05bcc05c6f0/lib/neo4j/railtie.rb:73:in `block (2 levels) in <class:Railtie>'
from /Users/jbrukh/.bundler/ruby/2.0.0/neo4j-a05bcc05c6f0/lib/neo4j/railtie.rb:72:in `each'
from /Users/jbrukh/.bundler/ruby/2.0.0/neo4j-a05bcc05c6f0/lib/neo4j/railtie.rb:72:in `block in <class:Railtie>'
from /Library/Ruby/Gems/2.0.0/gems/railties-4.1.6/lib/rails/initializable.rb:30:in `instance_exec'
from /Library/Ruby/Gems/2.0.0/gems/railties-4.1.6/lib/rails/initializable.rb:30:in `run'
from /Library/Ruby/Gems/2.0.0/gems/railties-4.1.6/lib/rails/initializable.rb:55:in `block in run_initializers'
from /System/Library/Frameworks/Ruby.framework/Versions/2.0/usr/lib/ruby/2.0.0/tsort.rb:150:in `block in tsort_each'
from /System/Library/Frameworks/Ruby.framework/Versions/2.0/usr/lib/ruby/2.0.0/tsort.rb:183:in `block (2 levels) in each_strongly_connected_component'
from /System/Library/Frameworks/Ruby.framework/Versions/2.0/usr/lib/ruby/2.0.0/tsort.rb:219:in `each_strongly_connected_component_from'
from /System/Library/Frameworks/Ruby.framework/Versions/2.0/usr/lib/ruby/2.0.0/tsort.rb:182:in `block in each_strongly_connected_component'
from /System/Library/Frameworks/Ruby.framework/Versions/2.0/usr/lib/ruby/2.0.0/tsort.rb:180:in `each'
from /System/Library/Frameworks/Ruby.framework/Versions/2.0/usr/lib/ruby/2.0.0/tsort.rb:180:in `each_strongly_connected_component'
from /System/Library/Frameworks/Ruby.framework/Versions/2.0/usr/lib/ruby/2.0.0/tsort.rb:148:in `tsort_each'
from /Library/Ruby/Gems/2.0.0/gems/railties-4.1.6/lib/rails/initializable.rb:54:in `run_initializers'
from /Library/Ruby/Gems/2.0.0/gems/railties-4.1.6/lib/rails/application.rb:300:in `initialize!'
from /Users/jbrukh/Triton/rails/triton-app/config/environment.rb:5:in `<top (required)>'
from /Library/Ruby/Gems/2.0.0/gems/activesupport-4.1.6/lib/active_support/dependencies.rb:247:in `require'
from /Library/Ruby/Gems/2.0.0/gems/activesupport-4.1.6/lib/active_support/dependencies.rb:247:in `block in require'
from /Library/Ruby/Gems/2.0.0/gems/activesupport-4.1.6/lib/active_support/dependencies.rb:232:in `load_dependency'
from /Library/Ruby/Gems/2.0.0/gems/activesupport-4.1.6/lib/active_support/dependencies.rb:247:in `require'
from /Library/Ruby/Gems/2.0.0/gems/spring-1.1.3/lib/spring/application.rb:92:in `preload'
from /Library/Ruby/Gems/2.0.0/gems/spring-1.1.3/lib/spring/application.rb:140:in `serve'
from /Library/Ruby/Gems/2.0.0/gems/spring-1.1.3/lib/spring/application.rb:128:in `block in run'
from /Library/Ruby/Gems/2.0.0/gems/spring-1.1.3/lib/spring/application.rb:122:in `loop'
from /Library/Ruby/Gems/2.0.0/gems/spring-1.1.3/lib/spring/application.rb:122:in `run'
from /Library/Ruby/Gems/2.0.0/gems/spring-1.1.3/lib/spring/application/boot.rb:18:in `<top (required)>'
from /Library/Ruby/Site/2.0.0/rubygems/core_ext/kernel_require.rb:54:in `require'
from /Library/Ruby/Site/2.0.0/rubygems/core_ext/kernel_require.rb:54:in `require'
from -e:1:in `<main>'

[tritonresearch]

Seems like it's the same issue, but now it's going to the substitute JSON parser.

irb(main):001:0> require 'json'
=> true
irb(main):002:0> require 'oj'
=> true
irb(main):003:0> require 'oj_mimic_json'
=> true
irb(main):004:0> JSON.parse({:a => 1})
ArgumentError: strict_parse() expected a String or IO Object.
from (irb):4:in `parse'
from (irb):4
from /usr/bin/irb:12:in `<main>'
irb(main):005:0> JSON.parse({:a => 1}.to_json)
=> {"a"=>1}

So the question for the maintainers are:

• Why is the response coming back as a Hash when serialized JSON is expected?
• Is this the right server response? The code looks for an errors field which is not present on this response.

[subvertallchris]

Good questions. I'm just about done what I'm working on and will see if I can reproduce it.

[subvertallchris]

I can't seem to reproduce it. Can you think of anything in your setup that would differ from a standard dev or test environment?

[subvertallchris]

I see you're using the username "jake." Where did you set that? I haven't seen anything in the docs about creating different accounts.

[tritonresearch]

The only thing that I can think of is that I have the patron gem, and Faraday is interoperable with it. However, upon removing patron, I still get the same error.

Can you add a check to see what the type of response.body is for your setup? For example, in cypher_authentication.rb, before line 40:

puts response.body.class.inspect

I would expect the body of a request to return binary data, but in my case it is returning a Hash.

[subvertallchris]

You mean auth_response, right? Mine is a String. Can you check that connection is a Faraday::Connection and response is Faraday::Response?

[tritonresearch]

Re: username. The way that I understand Neo4j to work is that it gives you a token and it pretty much ignores the username. I can show using curl that as long as the token is correct, the server authenticates correctly no matter what the username.

[tritonresearch]

Yep, checking. Thanks.

[tritonresearch]

Yep:

connection: Faraday::Connection response: Faraday::Response response.body: Hash

By the way, I am using:

• faraday (0.9.0)
• faraday_middleware (0.9.1)

[subvertallchris]

I was told the same thing about the username but when I give it anything other than "neo4j" as user, I get the error about invalid user/pass. Could you try with the default username?

I'm on the same versions of Faraday.

Do you have a proxy in between your machine and the server? Maybe the response is coming from it? That feels like a stretch... Check auth_response.headers[:server], it should be "Jetty(9.2.1.v20140609)".

[tritonresearch]

No effect from changing the username to "neo4j" :-(.

Can you post what your response looks like?

Even if the response that I'm getting came back as a string, it would still fail because there is no "errors" field on it. I would like to verify that this is the correct response altogether.

[subvertallchris]

You got it. That one comes back with => "{\n \"errors\" : [ {\n \"message\" : \"Invalid authorization token supplied.\",\n \"code\" : \"Neo.ClientError.Security.AuthorizationFailed\"\n } ]\n}".

[tritonresearch]

Yeah, when I corrupt my token, I get exactly the same response. The Faraday response comes back as a string, and everything works correctly.

Do you have a response from a successful connection?

[subvertallchris]

Oh oh oh, I see exactly what it's doing. It's actually giving you a valid response right away. This method expects the first connection attempt to either fail or give it the hint to show that auth isn't enabled, in which case it moves along. Yours already has a token -- that hash you posted at the beginning is a valid response after authentication. The question now is why it's trying to authenticate twice and where that first token is coming from.

[subvertallchris]

Aha! I reproduced it. One second here...

[tritonresearch]

Yeah, that sounds right. It looks like authenticate is trying to get a username/password pair to obtain a token. However, my understanding was that I should send the token as the password (maybe I am mistaken)?

Either way, the server seems to support this kind of token auth.

[subvertallchris]

That is what it's doing. I didn't implement support for just sending a token yet, it uses the username and password to request a token.

[subvertallchris]

But I think I have a fix for that... one sec!

[tritonresearch]

Sweet, glad we got to the root cause. Thanks Chris!

[tritonresearch]

Just for confirmation, I was able to connect correctly using the non-token credentials.

[subvertallchris]

Killer. I just need to write a spec and I'm pushing a fix that allows you to use the token, too.

[tritonresearch]

Thank you, sir!

[subvertallchris]

Can you try using the auth_token_response branch? I feel like this is going to need logic for handling unsuccessful responses but I'll mess with that later.

[tritonresearch]

Taking a look.

[tritonresearch]

It works!

[subvertallchris]

Awesome! I am sorry I didn't pick up on this right away, I was working on something else and a little distracted initially. It wasn't until I went to ask what your response looked like that I remembered you shared it at the beginning, then it all clicked.

[tritonresearch]

Also partially due to my lack of understanding of how Neo4j auth usually works. Glad we got it all sorted out. Also, great job on that Neo4j.rb presentation a few weeks ago!

Best, Jake

[subvertallchris]

The driver and it's authors' purpose is to know that for you! But I really appreciate you helping work through it. I implemented all the auth stuff quicklylast week based on the documentation, it was my first time seeing it, so I thought it pretty likely that I'd miss something.

Awesome, thanks! Really cool that you were there. Had you been using the gem prior to that?

[tritonresearch]

Yeah. We're prototyping a Neo4j backed product and I settled on Rails and neo4jrb, because of ActiveModel support. I think you guys have done an awesome job supporting it and providing features.

In general, it is a little scary running Neo4j. For one, it's not clear to me how well it scales because having 300K nodes tends to kill performance on medium AWS hardware, and we will have many more than that.

The other thing is that searckick is kind of squirrelly for searching through data. For instance, the reindexing poops out when the node data exceeds heap space and this operation becomes a challenge.

[subvertallchris]

It's supposed to handle millions of nodes and relationships pretty easily, though I've never had a large instance myself, I can only really echo back what I've heard. Did you tweak the cache sizes, adjust GC, do all that JVM stuff? That can change everything. Queries can often be adjusted to speed it up, too. Feel free to reach out if there's ever anything you'd like another set of eyes on.

Searchkick... yeah, I've had some problems with it, too. The guy who maintains it is really responsive to posted issues, from what I've seen. He might be able to help? There's also a lot of info out there about Elasticsearch performance tuning.

[subvertallchris]

This'll be merged into master sometime tomorrow, btw. I need to refactor how the responses are handled.

[tritonresearch]

I haven't done a deep dive into performance tuning, but I'll let you know how it goes. I'm just going off of observations of basic behavior.

Thanks again for your help, Chris.

[subvertallchris]

I think we're ready to merge this in! Reworked things under the hood a bit, everything is looking good. I'll delete the branch once you let me know you're good.