search

SeaQL / starfish-ql

✴️ An experimental graph database
https://www.sea-ql.org/StarfishQL/
GNU Affero General Public License v3.0
114 stars 3 forks source link

StarfishQL backed by multiple database backends #7

Closed billy1624 closed 2 years ago

billy1624 commented 2 years ago

TODO

billy1624 commented 2 years ago

Got an error when accessing /query endpoint on PostgreSQL

thread 'tokio-runtime-worker' panicked at 'u64 unsupported by sqlx-postgres', /Users/billy/.cargo/git/checkouts/sea-orm-62ec56b36bf9e642/134df4d/src/executor/query.rs:302:1
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
   >> Handler query panicked.
   >> A panic is treated as an internal server error.
   >> No 500 catcher registered. Using Rocket default.
billy1624 commented 2 years ago

Got an error when running freeport/backend on SQLite

Error Log ``` (base) starfish-ql/freeport/backend on  edge-collision [⇡!] is 📦 v0.1.0 via 🅒 base took 3s ➜ npm run restart > starfishql-backend@0.1.0 restart > rm -rf data && npm start > starfishql-backend@0.1.0 start > node . Repo found Metadata does not exist at data/meta. Resetting database... Commencing initial scrape... Removing existing data and metadata if any... Creating data folder... Creating intermediate data files... All file paths loaded. Data entries loading... 10000/81434 Data entries loading... 20000/81434 Data entries loading... 30000/81434 Data entries loading... 40000/81434 Data entries loading... 50000/81434 Data entries loading... 60000/81434 Data entries loading... 70000/81434 Data entries loading... 80000/81434 Data entries loading... 81434/81434 81434 data entries loaded from 81434 paths. 8 invalid edges filtered out: [ { from_node: 'chembasics', to_node: 'ptable' }, { from_node: 'essential', to_node: 'ptable' }, { from_node: 'rs-rocksdb-sys', to_node: 'rs-snappy-sys' }, { from_node: 's-rocksdb-sys', to_node: 's-snappy-sys' }, { from_node: 's-trie-standardmap', to_node: 'keccakhash' }, { from_node: 's-trie-standardmap', to_node: 's-bytes' }, { from_node: 'susy-rocksdb-sys', to_node: 'susy-snappy-sys' }, { from_node: 'susydev-rocksdb-sys', to_node: 'susydev-snappy-sys' } ] Collected 81434 nodes and 355414 edges. Released batch with 10000 items. Consuming nodes: 10000/81434 Released batch with 10000 items. Consuming nodes: 20000/81434 Released batch with 10000 items. Consuming nodes: 30000/81434 Released batch with 10000 items. Consuming nodes: 40000/81434 Released batch with 10000 items. Consuming nodes: 50000/81434 Released batch with 10000 items. Consuming nodes: 60000/81434 Released batch with 10000 items. Consuming nodes: 70000/81434 Released batch with 10000 items. Consuming nodes: 80000/81434 Released batch with 1434 items. Released batch with 10000 items. Consuming edges: 10000/355414 Released batch with 10000 items. Consuming edges: 20000/355414 Released batch with 10000 items. Consuming edges: 30000/355414 Released batch with 10000 items. Consuming edges: 40000/355414 Released batch with 10000 items. Consuming edges: 50000/355414 Released batch with 10000 items. Consuming edges: 60000/355414 Released batch with 10000 items. Consuming edges: 70000/355414 Released batch with 10000 items. Consuming edges: 80000/355414 Released batch with 10000 items. Consuming edges: 90000/355414 Released batch with 10000 items. Consuming edges: 100000/355414 Released batch with 10000 items. Consuming edges: 110000/355414 Released batch with 10000 items. Consuming edges: 120000/355414 Released batch with 10000 items. Consuming edges: 130000/355414 Released batch with 10000 items. Consuming edges: 140000/355414 Released batch with 10000 items. Consuming edges: 150000/355414 Released batch with 10000 items. Consuming edges: 160000/355414 Released batch with 10000 items. Consuming edges: 170000/355414 Released batch with 10000 items. Consuming edges: 180000/355414 Released batch with 10000 items. Consuming edges: 190000/355414 Released batch with 10000 items. Consuming edges: 200000/355414 Released batch with 10000 items. Consuming edges: 210000/355414 Released batch with 10000 items. Consuming edges: 220000/355414 Released batch with 10000 items. Consuming edges: 230000/355414 Released batch with 10000 items. Consuming edges: 240000/355414 Released batch with 10000 items. Consuming edges: 250000/355414 Released batch with 10000 items. Consuming edges: 260000/355414 Released batch with 10000 items. Consuming edges: 270000/355414 Released batch with 10000 items. Consuming edges: 280000/355414 Released batch with 10000 items. Consuming edges: 290000/355414 Released batch with 10000 items. Consuming edges: 300000/355414 Released batch with 10000 items. Consuming edges: 310000/355414 Released batch with 10000 items. Consuming edges: 320000/355414 Released batch with 10000 items. Consuming edges: 330000/355414 Released batch with 10000 items. Consuming edges: 340000/355414 Released batch with 10000 items. Consuming edges: 350000/355414 Released batch with 5414 items. Inserted 436848 items into database in 121s with 0 errors caught. Calculating connectivities... Error: Request failed with status code 500 at createError (/Applications/MAMP/htdocs/starfish-ql/freeport/backend/node_modules/axios/lib/core/createError.js:16:15) at settle (/Applications/MAMP/htdocs/starfish-ql/freeport/backend/node_modules/axios/lib/core/settle.js:17:12) at IncomingMessage.handleStreamEnd (/Applications/MAMP/htdocs/starfish-ql/freeport/backend/node_modules/axios/lib/adapters/http.js:293:11) at IncomingMessage.emit (node:events:402:35) at endReadableNT (node:internal/streams/readable:1343:12) at processTicksAndRejections (node:internal/process/task_queues:83:21) { config: { transitional: { silentJSONParsing: true, forcedJSONParsing: true, clarifyTimeoutError: false }, adapter: [Function: httpAdapter], transformRequest: [ [Function: transformRequest] ], transformResponse: [ [Function: transformResponse] ], timeout: 0, xsrfCookieName: 'XSRF-TOKEN', xsrfHeaderName: 'X-XSRF-TOKEN', maxContentLength: -1, maxBodyLength: -1, validateStatus: [Function: validateStatus], headers: { Accept: 'application/json, text/plain, */*', 'Content-Type': 'application/json', 'User-Agent': 'axios/0.24.0', 'Content-Length': 24 }, method: 'post', url: 'http://127.0.0.1:8000/mutate?', data: '{"cal_conn":["depends"]}' }, request: ClientRequest { _events: [Object: null prototype] { abort: [Function (anonymous)], aborted: [Function (anonymous)], connect: [Function (anonymous)], error: [Function (anonymous)], socket: [Function (anonymous)], timeout: [Function (anonymous)], prefinish: [Function: requestOnPrefinish] }, _eventsCount: 7, _maxListeners: undefined, outputData: [], outputSize: 0, writable: true, destroyed: false, _last: true, chunkedEncoding: false, shouldKeepAlive: false, maxRequestsOnConnectionReached: false, _defaultKeepAlive: true, useChunkedEncodingByDefault: true, sendDate: false, _removedConnection: false, _removedContLen: false, _removedTE: false, _contentLength: null, _hasBody: true, _trailer: '', finished: true, _headerSent: true, _closed: false, socket: Socket { connecting: false, _hadError: false, _parent: null, _host: null, _readableState: [ReadableState], _events: [Object: null prototype], _eventsCount: 7, _maxListeners: undefined, _writableState: [WritableState], allowHalfOpen: false, _sockname: null, _pendingData: null, _pendingEncoding: '', server: null, _server: null, parser: null, _httpMessage: [Circular *1], [Symbol(async_id_symbol)]: 902658, [Symbol(kHandle)]: [TCP], [Symbol(kSetNoDelay)]: false, [Symbol(lastWriteQueueSize)]: 0, [Symbol(timeout)]: null, [Symbol(kBuffer)]: null, [Symbol(kBufferCb)]: null, [Symbol(kBufferGen)]: null, [Symbol(kCapture)]: false, [Symbol(kBytesRead)]: 0, [Symbol(kBytesWritten)]: 0, [Symbol(RequestTimeout)]: undefined }, _header: 'POST /mutate? HTTP/1.1\r\n' + 'Accept: application/json, text/plain, */*\r\n' + 'Content-Type: application/json\r\n' + 'User-Agent: axios/0.24.0\r\n' + 'Content-Length: 24\r\n' + 'Host: 127.0.0.1:8000\r\n' + 'Connection: close\r\n' + '\r\n', _keepAliveTimeout: 0, _onPendingData: [Function: nop], agent: Agent { _events: [Object: null prototype], _eventsCount: 2, _maxListeners: undefined, defaultPort: 80, protocol: 'http:', options: [Object: null prototype], requests: [Object: null prototype] {}, sockets: [Object: null prototype], freeSockets: [Object: null prototype] {}, keepAliveMsecs: 1000, keepAlive: false, maxSockets: Infinity, maxFreeSockets: 256, scheduling: 'lifo', maxTotalSockets: Infinity, totalSocketCount: 1, [Symbol(kCapture)]: false }, socketPath: undefined, method: 'POST', maxHeaderSize: undefined, insecureHTTPParser: undefined, path: '/mutate?', _ended: true, res: IncomingMessage { _readableState: [ReadableState], _events: [Object: null prototype], _eventsCount: 3, _maxListeners: undefined, socket: [Socket], httpVersionMajor: 1, httpVersionMinor: 1, httpVersion: '1.1', complete: true, rawHeaders: [Array], rawTrailers: [], aborted: false, upgrade: false, url: '', method: null, statusCode: 500, statusMessage: 'Internal Server Error', client: [Socket], _consuming: false, _dumped: false, req: [Circular *1], responseUrl: 'http://127.0.0.1:8000/mutate?', redirects: [], [Symbol(kCapture)]: false, [Symbol(kHeaders)]: [Object], [Symbol(kHeadersCount)]: 18, [Symbol(kTrailers)]: null, [Symbol(kTrailersCount)]: 0, [Symbol(RequestTimeout)]: undefined }, aborted: false, timeoutCb: null, upgradeOrConnect: false, parser: null, maxHeadersCount: null, reusedSocket: false, host: '127.0.0.1', protocol: 'http:', _redirectable: Writable { _writableState: [WritableState], _events: [Object: null prototype], _eventsCount: 2, _maxListeners: undefined, _options: [Object], _ended: true, _ending: true, _redirectCount: 0, _redirects: [], _requestBodyLength: 24, _requestBodyBuffers: [], _onNativeResponse: [Function (anonymous)], _currentRequest: [Circular *1], _currentUrl: 'http://127.0.0.1:8000/mutate?', [Symbol(kCapture)]: false }, [Symbol(kCapture)]: false, [Symbol(kNeedDrain)]: false, [Symbol(corked)]: 0, [Symbol(kOutHeaders)]: [Object: null prototype] { accept: [Array], 'content-type': [Array], 'user-agent': [Array], 'content-length': [Array], host: [Array] } }, response: { status: 500, statusText: 'Internal Server Error', headers: { 'content-type': 'application/json', server: 'Rocket', 'x-frame-options': 'SAMEORIGIN', 'permissions-policy': 'interest-cohort=()', 'x-content-type-options': 'nosniff', 'access-control-allow-origin': '*', 'access-control-allow-methods': 'GET, POST, PATCH, OPTIONS', 'content-length': '69', date: 'Thu, 21 Apr 2022 06:49:04 GMT' }, config: { transitional: [Object], adapter: [Function: httpAdapter], transformRequest: [Array], transformResponse: [Array], timeout: 0, xsrfCookieName: 'XSRF-TOKEN', xsrfHeaderName: 'X-XSRF-TOKEN', maxContentLength: -1, maxBodyLength: -1, validateStatus: [Function: validateStatus], headers: [Object], method: 'post', url: 'http://127.0.0.1:8000/mutate?', data: '{"cal_conn":["depends"]}' }, request: ClientRequest { _events: [Object: null prototype], _eventsCount: 7, _maxListeners: undefined, outputData: [], outputSize: 0, writable: true, destroyed: false, _last: true, chunkedEncoding: false, shouldKeepAlive: false, maxRequestsOnConnectionReached: false, _defaultKeepAlive: true, useChunkedEncodingByDefault: true, sendDate: false, _removedConnection: false, _removedContLen: false, _removedTE: false, _contentLength: null, _hasBody: true, _trailer: '', finished: true, _headerSent: true, _closed: false, socket: [Socket], _header: 'POST /mutate? HTTP/1.1\r\n' + 'Accept: application/json, text/plain, */*\r\n' + 'Content-Type: application/json\r\n' + 'User-Agent: axios/0.24.0\r\n' + 'Content-Length: 24\r\n' + 'Host: 127.0.0.1:8000\r\n' + 'Connection: close\r\n' + '\r\n', _keepAliveTimeout: 0, _onPendingData: [Function: nop], agent: [Agent], socketPath: undefined, method: 'POST', maxHeaderSize: undefined, insecureHTTPParser: undefined, path: '/mutate?', _ended: true, res: [IncomingMessage], aborted: false, timeoutCb: null, upgradeOrConnect: false, parser: null, maxHeadersCount: null, reusedSocket: false, host: '127.0.0.1', protocol: 'http:', _redirectable: [Writable], [Symbol(kCapture)]: false, [Symbol(kNeedDrain)]: false, [Symbol(corked)]: 0, [Symbol(kOutHeaders)]: [Object: null prototype] }, data: 'Execution Error: error returned from database: too many SQL variables' }, isAxiosError: true, toJSON: [Function: toJSON] } ```
billy1624 commented 2 years ago

@shpun817 please starts from this branch, https://github.com/SeaQL/starfish-ql/tree/db-backends