search

snowflakedb / snowflake-connector-nodejs

NodeJS driver
Apache License 2.0
124 stars 132 forks source link

Upgrade snowflake-sdk from 1.6.23 to 1.9.2 - Connection.execute fails with `axios.request is not a function` #730

Closed melindalimafresh closed 10 months ago

melindalimafresh commented 11 months ago

Please answer these questions before submitting your issue. In order to accurately debug the issue this information is required. Thanks!

  1. What version of NodeJS driver are you using? 1.9.2

  2. What operating system and processor architecture are you using? OS: MacOS Ventura 13.5.2 Chip: Apple M1 Pro

  3. What version of NodeJS are you using? (node --version and npm --version) v18.17.1

  4. What are the component versions in the environment (npm list)?

├── @azure/cosmos@3.15.1 ├── @azure/identity@3.2.3 ├── @azure/service-bus@7.5.1 ├── @azure/storage-blob@12.9.0 ├── @babel/cli@7.21.5 ├── @babel/core@7.22.1 ├── @babel/node@7.22.1 ├── @babel/plugin-proposal-class-properties@7.18.6 ├── @babel/plugin-proposal-nullish-coalescing-operator@7.18.6 ├── @babel/plugin-proposal-optional-chaining@7.21.0 ├── @babel/plugin-syntax-numeric-separator@7.10.4 ├── @babel/plugin-transform-typescript@7.22.0 ├── @babel/preset-env@7.22.2 ├── @babel/preset-typescript@7.21.5 ├── @envelop/core@4.0.0 ├── @fast-csv/parse@4.3.6 ├── @godaddy/terminus@4.9.0 ├── @graphql-codegen/cli@4.0.1 ├── @graphql-codegen/typescript-resolvers@2.1.1 ├── @graphql-codegen/typescript@2.1.1 ├── @graphql-tools/schema@10.0.0 ├── @graphql-tools/utils@10.0.6 ├── @jest-mock/express@1.4.5 ├── @jest/types@29.1.2 ├── @octokit/graphql@5.0.5 ├── @sentry/cli@2.5.0 ├── @sentry/node@6.19.7 ├── @sentry/utils@7.64.0 ├── @supercharge/promise-pool@2.1.0 ├── @types/bcrypt-nodejs@0.0.31 ├── @types/chalk@2.2.0 ├── @types/commander@2.12.2 ├── @types/connect-datadog@0.0.6 ├── @types/connect-redis@0.0.20 ├── @types/cron@1.7.3 ├── @types/express-session@1.17.4 ├── @types/express@4.17.13 ├── @types/generic-pool@3.1.11 ├── @types/google-spreadsheet@3.1.5 ├── @types/graphql-upload@15.0.2 ├── @types/ioredis-mock@8.2.2 ├── @types/jest@29.1.1 ├── @types/jsonwebtoken@8.5.0 ├── @types/lodash@4.14.162 ├── @types/mjml@4.7.0 ├── @types/moo@0.5.5 ├── @types/multer@1.4.7 ├── @types/nearley@2.11.2 ├── @types/node@20.10.3 ├── @types/pg@8.6.5 ├── @types/redis@2.8.31 ├── @types/rox-node@5.0.1 ├── @types/semver@7.5.1 ├── @types/serve-static@1.13.9 ├── @types/sinon@9.0.10 ├── @types/snowflake-sdk@1.6.7 ├── @types/supertest@2.0.10 ├── @types/uuid@8.3.0 ├── @types/validator@13.1.0 ├── @types/yesql@4.1.1 ├── @typescript-eslint/eslint-plugin@5.17.0 ├── @typescript-eslint/parser@5.17.0 ├── apollo-server-core@3.12.1 ├── apollo-server-express@3.6.7 ├── apollo-server-plugin-base@3.7.2 ├── apollo-server@3.6.7 ├── axios@1.6.0 ├── babel-jest@29.5.0 ├── babel-plugin-module-resolver@5.0.0 ├── bcrypt-nodejs@0.0.3 ├── calendar-date@2.2.1 ├── celery-node@0.5.9 ├── chalk@4.1.2 ├── commander@9.3.0 ├── connect-datadog@0.0.9 ├── connect-redis@6.0.0 ├── cors@2.8.5 ├── cron@1.8.2 ├── csv-parse@5.2.0 ├── dataloader@2.0.0 ├── dd-trace@4.6.0 ├── decimal.js@10.4.3 ├── dedent@0.7.0 ├── deep-object-diff@1.1.9 ├── dotenv@8.2.0 ├── email-validator@2.0.4 ├── env-cmd@10.1.0 ├── eslint-config-airbnb-base@15.0.0 ├── eslint-config-prettier@6.9.0 ├── eslint-import-resolver-babel-module@5.3.2 ├── eslint-plugin-import@2.27.5 ├── eslint-plugin-jest@23.20.0 ├── eslint-plugin-prettier@3.1.2 ├── eslint@7.32.0 ├── exceljs@4.3.0 ├── exponential-backoff@3.1.0 ├── express-rate-limit@6.3.0 ├── express-session@1.17.2 ├── express@4.18.2 ├── fast-csv@4.3.6 ├── fast-safe-stringify@2.0.7 ├── form-data@4.0.0 ├── generic-pool@3.8.2 ├── google-auth-library@9.4.1 ├── google-spreadsheet@4.1.1 ├── grant@5.4.15 ├── graphql-middleware@6.1.4 ├── graphql-request@4.2.0 ├── graphql-upload@15.0.2 ├── graphql@15.5.1 ├── hot-shots@8.1.0 ├── husky@7.0.4 ├── ioredis-mock@8.8.1 ├── ioredis@5.3.2 ├── jest-junit@13.0.0 ├── jest-mock-axios@4.5.0 ├── jest@29.1.2 ├── jsonwebtoken@9.0.0 ├── lint-prepush@2.2.1 ├── lodash@4.17.21 ├── mjml@4.11.0 ├── mockdate@3.0.5 ├── moment-timezone@0.5.35 ├── moo@0.5.1 ├── ms@2.1.3 ├── multer@1.4.5-lts.1 ├── nanoid@3.1.31 ├── nearley@2.20.1 ├── nodemon@3.0.1 ├── openai@3.3.0 ├── patch-package@6.4.7 ├── pdf.js-extract@0.2.1 ├── pg@8.4.1 ├── plop@3.1.2 ├── prettier@1.19.1 ├── readline-sync@1.4.10 ├── request-logs@2.1.4 ├── rox-node@5.4.8 ├── semver@7.5.4 ├── sequelize@6.31.1 ├── sinon@9.2.3 ├── snapshot-diff@0.9.0 ├── snowflake-sdk@1.9.1 ├── sort-package-json@2.4.1 ├── sql-template-strings@2.2.2 ├── supertest@6.1.6 ├── ts-node@10.7.0 ├── tsconfig-paths@4.1.0 ├── typedoc@0.22.17 ├── typescript@4.6.3 ├── uuid@8.3.2 ├── winston-transport@4.4.0 ├── winston@3.3.3 ├── yargs@17.5.1 └── yesql@5.0.0

5.Server version:* E.g. 1.90.1 You may get the server version by running a query:

SELECT CURRENT_VERSION();
  1. What did you do?
    • Updated Snowflake-sdk version from 1.6.23 to 1.9.2
    • yarn build && yarn install
    • Create the Snowflake pool with createPool.
    • Call execute with SQL parameters CREATE DATABASE IF NOT EXISTS fails with the error below.
TypeError: axios.request is not a function

      at NodeHttpClient.sendRequest (node_modules/snowflake-sdk/lib/http/base.js:41:21)

  1. What did you expect to see?

    I expected this to create the db and not fail due to an axios import issue.

  2. Can you set logging to DEBUG and collect the logs?

    https://community.snowflake.com/s/article/How-to-generate-log-file-on-Snowflake-connectors

    e.g Add this to get standard output.

var snowflake = require('snowflake-sdk');
snowflake.configure(
{
  logLevel: 'trace'
});
  1. What is your Snowflake account identifier, if any? (Optional)
sfc-gh-dszmolka commented 11 months ago

thank you for submitting this issue; we'll take a look.

sfc-gh-dszmolka commented 11 months ago

i'm using following script to try and reproduce the issue:

# cat pool.js 
var  snowflake = require('snowflake-sdk');

console.log(`\nNode.js process version is: ${process.version}\n`);

snowflake.configure({ logLevel : 'trace'});

const account = process.env.SFACCOUNT;
const username = process.env.SFUSER;
const password = process.env.SFPASS;
const warehouse = process.env.SFWH;
const application = 'Test-GH-730';

const poolSize = 3;

const pool = snowflake.createPool(
    {
      account: account,
      password: password,
      username: username,
      warehouse: warehouse,
      application: application,
      timeout: 30_000,
    },
    {
      evictionRunIntervalMillis: 30_000,
      idleTimeoutMillis: 30_000,
      max: poolSize,
      min: 1,
    },
  );

pool.use(async (clientConnection) =>
{
    const statement = await clientConnection.execute({
        sqlText: 'CREATE DATABASE IF NOT EXISTS GH730;',
        complete: function (err, stmt, rows)
        {
            var stream = stmt.streamRows();
            stream.on('data', function (row)
            {
                console.log(row);
            });
            stream.on('end', function (row)
            {
                console.log('All rows consumed');
            });
        }
    });
})
;

with axios 1.6.0 (by default 1.6.2 is installed)

# npm list axios
node@ /node
+-- axios@1.6.0
`-- snowflake-sdk@1.9.2
  `-- axios@1.6.0 deduped

# npm list
node@ /node
+-- axios@1.6.0
+-- log-timestamp@0.3.0
`-- snowflake-sdk@1.9.2

edit --> in your npm list output, snowflake-sdk is at 1.9.1, but you reported issue against snowflake-sdk 1.9.2, can you please confirm which version you're having problem with ?

and looks to be executing correctly to me:

# node pool.js 

Node.js process version is: v18.16.1

{"level":"DEBUG","message":"[6:36:26.406 AM]: 300"}
{"level":"DEBUG","message":"[6:36:26.415 AM]: Contacting SF: /session/v1/login-request?requestId=[..], (1/7)"}
..
{ status: 'Database GH730 successfully created.' }
All rows consumed

with no error whatsoever.

base.js was indeed recently fixed with 1.9.2 (https://github.com/snowflakedb/snowflake-connector-nodejs/pull/713) but I'm wondering how you can get to the error.

would it be please possible to send a runnable reproduction which when executed leads to the issue you're seeing? this would be very helpful in debugging. thank you in advance !

edit installed the same modules from your npm list and my test script still runs correctly, so it would really be important to have a reproduction which throws the error for you.

sscraigie commented 11 months ago

I think i got a similar error:

ReferenceError: request is not defined
     at NodeHttpClient.sendRequest (webpack-internal:///(rsc)/./node_modules/snowflake-sdk/lib/http/base.js:32:17)
sfc-gh-dszmolka commented 11 months ago

Until now, the issue does not reproduce for me yet, but I would really love to take a look at it, so if anyone is able to create a reproducible scenario; please do share - thank you in advance !

Satevg commented 11 months ago

Not sure if it's related, but after update from 1.6.x to 1.9.2, we've got failed tests:

Summary of all failing tests
 FAIL  src/common/services/SnowflakeService.spec.ts
  ● Test suite failed to run

    Jest encountered an unexpected token

    Jest failed to parse a file. This happens e.g. when your code or its dependencies use non-standard JavaScript syntax, or when Jest is not configured to support such syntax.

    Out of the box Jest supports Babel, which will be used to transform your files into valid JS based on your Babel configuration.

    By default "node_modules" folder is ignored by transformers.

    Here's what you can do:
     • If you are trying to use ECMAScript Modules, see https://jestjs.io/docs/ecmascript-modules for how to enable it.
     • If you are trying to use TypeScript, see https://jestjs.io/docs/getting-started#using-typescript
     • To have some of your "node_modules" files transformed, you can specify a custom "transformIgnorePatterns" in your config.
     • If you need a custom transformation specify a "transform" option in your config.
     • If you simply want to mock your non-JS modules (e.g. binary assets) you can stub them out with the "moduleNameMapper" config option.

    You'll find more details and examples of these config options in the docs:
    https://jestjs.io/docs/configuration
    For information about custom transformations, see:
    https://jestjs.io/docs/code-transformation

    Details:

    /home/satevg/Dev/aquicore/microservices/node_modules/snowflake-sdk/node_modules/axios/index.js:1
    ({"Object.<anonymous>":function(module,exports,require,__dirname,__filename,jest){import axios from './lib/axios.js';
sfc-gh-dszmolka commented 11 months ago

tried following scenario as well.

  1. install axios 1.6.0
  2. install snowflake-sdk@1.6.23 
    
# npm why axios
axios@1.6.0
node_modules/axios
axios@"^1.6.0" from the root project

axios@0.27.2 node_modules/snowflake-sdk/node_modules/axios axios@"^0.27.2" from snowflake-sdk@1.6.23 node_modules/snowflake-sdk snowflake-sdk@"^1.6.23" from the root project

3. run above test script, confirm it works
4. install snowflake-sdk@1.9.2

npm why axios

axios@1.6.0 node_modules/axios axios@"^1.6.0" from the root project axios@"^1.6.0" from snowflake-sdk@1.9.2 node_modules/snowflake-sdk snowflake-sdk@"^1.9.2" from the root project


5. run above test scripts again. still works

probably something is broken ,but until now i'm unable to reproduce and get to the same issue. 

But since it is broken for at least for 3 of you guys unfortunately - if any of you happens to have any closer details besides the fact it breaks for you after upgrade, that would be really helpful.
sfc-gh-dszmolka commented 10 months ago

issue seems to be not generically fitting for all use cases and there has been no reproducible scenario for a long while. i'm now going to mark this issue as closed for now but in case any of you can share a reproduction scenario which leads to this issue, kindly do so and we can investigate further.