wujjpp commented 2 years ago

Nebula Nodejs SDK

This repository provides Nebula client API in Nodejs.

Features

Muti-Server Support
Auto-reconnection support

Client will try to reconnect forever, until the server is available again.
Connection pool support
Disconnection detection

A heartbeat mechanism is implemented, client will send ping to server each pingInterval ms for detect connective
Thrift enhancement

fix auto reconnect issue#2407

fix performance issue in huge data scene#2483

API

Connection Options

parameter	type	description
servers	string[]	nebula servers
userName	string	username for login
password	string	password for login
poolSize	number	Pool size for each server(Optional, default：5)
bufferSize	number	Command cache in offline or before established connect (Optional, defaul: 2000)
executeTimeout	number	Command executing timeout in ms (Optional, default：10000)
pingInterval	number	for keepalive, ping duration in ms, (Optional, default：60000）

How To

Install

For compiling C++ native module, node-gyp is required, you can install node-gyp by npm install -g node-gyp

npm install nebula-nodejs --save --unsafe-perm

Simple and convenient API

// ESM
import { createClient } from 'nebula-nodejs'

// CommonJS
// const { createClient } = require('nebula-nodejs')

// Connection Options
const options = {
  servers: ['ip-1:port','ip-2:port'],
  userName: 'xxx',
  password: 'xxx',
  database: 'database name',
  poolSize: 5,
  bufferSize: 2000,
  executeTimeout: 15000,
  pingInterval: 60000
}

// Create client
const client = createClient(options)

// Execute command
// 1. return parsed data (recommend)
const response = await client.execute('GET SUBGRAPH 3 STEPS FROM -7897618527020261406')
// 2. return nebula original data
const responseOriginal = await client.execute('GET SUBGRAPH 3 STEPS FROM -7897618527020261406', true)

Events

parameter	description
sender	the individual connection in connection pool
error	Nebula Error
retryInfo	Retry information
retryInfo.delay	delay time
retryInfo.attempt	total attempts

const client = createClient(options)

// connection is ready for executing command
client.on('ready', ({sender}) => {

})

// error occurs
client.on('error', ({ sender, error }) => {

})

// connected event
client.on('connected', ({ sender }) => {

})

// authorized successfully
client.on('authorized', ({ sender }) => {

})

// reconnecting
client.on('reconnecting', ({ sender, retryInfo }) => {

})

// closed
client.on('close', { sender }) => {

}

About hash64 function

nebula-nodejs exports hash64 function for converting string to string[], it's based on MurmurHash3.

// ESM
import { hash64 } from 'nebula-nodejs'

// CommonJS
// const { hash64 } = require('nebula-nodejs')

const results = hash64('f10011b64aa4e7503cd45a7fdc24387b')

console.log(results)

// Output:
// ['2852836996923339651', '-6853534673140605817']

About Int64

nodejs cannot repreent Int64, so we convert Int64 bytes to string

// ESM
import { bytesToLongLongString } from 'nebula-nodejs'

// CommonJS
// const { bytesToLongLongString } = require('nebula-nodejs')

const s = '-7897618527020261406'

const buffer = [146, 102, 5, 203, 5, 105, 223, 226]
const result = bytesToLongLongString(buffer)

// result equals s

Development

Build

git clone https://github.com/vesoft-inc/nebula-node.git
cd nebula-node
npm install --unsafe-perm
npm run build

Unit Test

npm run build
npm run test

Unit Test Coverage

npm run coverage

Publish

npm run build
cd dist
npm publish

TODO

Not implemented data type for auto parser

Data Type	property name in nebula response
NMap	mVal
NSet	uVal
DataSet	gVal

CLAassistant commented 2 years ago

All committers have signed the CLA.

Reid00 commented 2 years ago

@wey-gu here is the pr, pls take a look.

wey-gu commented 2 years ago

@wey-gu here is the pr, pls take a look.

Thank you @Reid00 @wujjpp! Great Job! You even fixed two issues on upstream apache thrift 👍🏻. We will look into it.

Best Regards, Wey

wujjpp commented 2 years ago

@wey-gu 这个PR还合吗？

nianiaJR commented 2 years ago

We'll deal with it in this week, thanks for your following🤝

wey-gu commented 2 years ago

@wey-gu 这个PR还合吗？

Sorry for the late response, sure we will, hopefully it could be done in short time as aligned with @nianiaJR

wujjpp commented 2 years ago

@wujjpp Is there any plan to make up the unit test/example in future?

I find it only like a todo demo about test?

I have a full test case located at tests/nebula.test-disabled, but I didn't have nebula enviroment for testing in github env. So, I rename nebula.test.ts to nebula.test-disabled, you can revert its name and add unit tests for testing. BTW, those test cases are based on our nebula production server.

/* eslint-disable max-len */
/**
 * Created by Wu Jian Ping on - 2021/06/09.
 */

import { expect } from 'chai'
import createClient, { ClientOption } from '../dist'

const relationServer: ClientOption = {
  servers: ['10.0.1.100:9669', '10.0.1.101:9669', '10.0.1.102:9669'],
  userName: 'xxxxx',
  space: 'xxxxx',
  database: 'partion1',
  pingInterval: 5000,
  poolSize: 2
}

const riskInfoServer: ClientOption = {
  servers: ['10.0.1.100:9669', '10.0.1.101:9669', '10.0.1.102:9669'],
  userName: 'xxxxx',
  space: 'xxxxx',
  database: 'partion2',
  pingInterval: 5000,
  poolSize: 2
}

const commands = {
  cmd1: 'GET SUBGRAPH 2 STEPS FROM -7897618527020261406',
  cmd2: 'fetch prop on Company -7897618527020261406',
  cmd3: 'go from 815677140545765099 over Relation yield Relation._src as src, Relation._dst as dst, Relation.relation_name as relation_name, Relation.prop1 as prop1, Relation.prop2 as prop2, Relation.prop3 as prop3, Relation.prop4 as prop4, $^.Person.name as src_p_name, $^.Company.name as src_c_name,  $$.Person.name as dst_p_name, $$.Company.name as dst_c_name | limit 1000',
  cmd4: 'find noloop path from -4075961126534726064 to 815677140545765099 over Relation bidirect upto 2 steps',
  cmd5: 'fetch prop on Relation -4075961126534726064->815677140545765099@-5101473608195470769'
}

describe('nebula', () => {
  // eslint-disable-next-line prefer-arrow/prefer-arrow-functions
  it('test-case-1', async () => {
    const client = createClient(relationServer)

    const response1 = await client.execute(commands.cmd1)

    expect(response1.data?._vertices?.length).greaterThan(0)
    expect(response1.data?._edges?.length).greaterThan(0)

    await client.close()
  })

  it('test-case-2', async () => {
    const client = createClient(relationServer)

    const response1 = await client.execute(commands.cmd2)

    expect(response1.data?.vertices_?.length).greaterThan(0)

    await client.close()
  })

  it('test-case-3', async () => {
    const client = createClient(riskInfoServer)

    const response1 = await client.execute(commands.cmd3)
    expect(response1.data?.src?.length).greaterThan(0)

    await client.close()
  })

  it('test-case-4', async () => {
    const client = createClient(riskInfoServer)

    const response1 = await client.execute(commands.cmd4)

    expect(response1.data?.path?.length).greaterThan(0)

    await client.close()
  })

  it('test-case-5', async () => {
    const client = createClient(riskInfoServer)

    const response1 = await client.execute(commands.cmd5)

    expect(response1.data?.edges_?.length).greaterThan(0)

    await client.close()
  })

  it('test-promise-all', async () => {
    const client = createClient(relationServer)
    const [resp1, resp2, resp3] = await Promise.all([
      client.execute('GET SUBGRAPH 1 STEPS FROM -7897618527020261406', false),
      client.execute('GET SUBGRAPH 2 STEPS FROM -7897618527020261406', false),
      client.execute('GET SUBGRAPH 3 STEPS FROM -7897618527020261406', false)
    ])

    console.log(`resp11:${resp1.data._vertices.length}, ${resp1.data._edges.length}`) // eslint-disable-line
    console.log(`resp21:${resp2.data._vertices.length}, ${resp2.data._edges.length}`) // eslint-disable-line
    console.log(`resp31:${resp3.data._vertices.length}, ${resp3.data._edges.length}`) // eslint-disable-line

    await client.close()
  })

  after(async () => {
    process.exit()
  })
})

nebula-contrib / nebula-node