Artrois / Google_Translator_API

Scripts to parse PO and YML files and push it to google translator and then store the results
MIT License
0 stars 0 forks source link

is possible translate to es, de and it this file? #1

Open johnfelipe opened 1 year ago

johnfelipe commented 1 year ago

https://gist.github.com/1641f7402fdfc3969e9edb7f08bccc8e

johnfelipe commented 1 year ago
root@ubuntu20virgeninwindows:~/Google_Translator_API# node .\translate_PO_YML.js ./sdg.yml
node:internal/modules/cjs/loader:936
  throw err;
  ^

Error: Cannot find module '/root/Google_Translator_API/.translate_PO_YML.js'
    at Function.Module._resolveFilename (node:internal/modules/cjs/loader:933:15)
    at Function.Module._load (node:internal/modules/cjs/loader:778:27)
    at Function.executeUserEntryPoint [as runMain] (node:internal/modules/run_main:77:12)
    at node:internal/main/run_main_module:17:47 {
  code: 'MODULE_NOT_FOUND',
  requireStack: []
}

but something happend

root@ubuntu20virgeninwindows:~/Google_Translator_API# node get_supported_langs.js
/root/Google_Translator_API/node_modules/google-auth-library/build/src/auth/googleauth.js:184
            throw new Error('Could not load the default credentials. Browse to https://cloud.google.com/docs/authentication/getting-started for more information.');
                  ^

Error: Could not load the default credentials. Browse to https://cloud.google.com/docs/authentication/getting-started for more information.
    at GoogleAuth.getApplicationDefaultAsync (/root/Google_Translator_API/node_modules/google-auth-library/build/src/auth/googleauth.js:184:19)
    at processTicksAndRejections (node:internal/process/task_queues:96:5)
    at async GoogleAuth.getClient (/root/Google_Translator_API/node_modules/google-auth-library/build/src/auth/googleauth.js:610:17)
    at async GrpcClient._getCredentials (/root/Google_Translator_API/node_modules/google-gax/build/src/grpc.js:145:24)
    at async GrpcClient.createStub (/root/Google_Translator_API/node_modules/google-gax/build/src/grpc.js:308:23)

i have in root file traducciones-352415-79659df1c6f1.json, downloaded for my translator account in GCP

how solve this?

Artrois commented 1 year ago

Hi, for the first issue i think you can run like this: node translate_PO_YML.js sdg.yml Further, you need to modify these variables in translate_strings_with_googleAPI.js: const projectId = ''; const location = 'global'; const source_lang = 'de'; let target_lang = 'en';

For the second issue with get_supported_langs.js you need to modify: const projectId = ''; const location = 'global'; Basically put your google project ID and make sure google translate API is activated with your projectID.

Artrois commented 1 year ago

also be aware of limitations imposed by google translate API for daily requests and requests per minute. This script can shoot out huge bunch of strings at once and reach the limits in a heartbeat.

Artrois commented 1 year ago

I just checked your yml file and i can tell you for sure my script wont work with your yml file as it is formatted differently compared to my implementation. My script is expecting: " - text:" or " - " as prefix for translations.

johnfelipe commented 1 year ago

node .\translate_PO_YML.js ./sdg.yml

can u edit section for regex and then try?

johnfelipe commented 1 year ago

Hi, for the first issue i think you can run like this: node translate_PO_YML.js sdg.yml Further, you need to modify these variables in translate_strings_with_googleAPI.js: const projectId = ''; const location = 'global'; const source_lang = 'de'; let target_lang = 'en';

For the second issue with get_supported_langs.js you need to modify: const projectId = ''; const location = 'global'; Basically put your google project ID and make sure google translate API is activated with your projectID.

still not works SNAG-0059

SNAG-0060

SNAG-0061 SNAG-0062

Artrois commented 1 year ago

The script uses Google Cloud Translation library. Did you set env properly? To use client libs put path to key.json to env path: PowerShell $env:GOOGLE_APPLICATION_CREDENTIALS="KEY_PATH"

CMD set GOOGLE_APPLICATION_CREDENTIALS=KEY_PATH

johnfelipe commented 1 year ago

And with putty for Ubuntu?

El mié, 3 de ago. de 2022, 10:40 a. m., Artrois @.***> escribió:

The script uses Google Cloud Translation library. Did you set env properly? To use client libs put path to key.json to env path: PowerShell $env:GOOGLE_APPLICATION_CREDENTIALS="KEY_PATH"

CMD set GOOGLE_APPLICATION_CREDENTIALS=KEY_PATH

— Reply to this email directly, view it on GitHub https://github.com/Artrois/Google_Translator_API/issues/1#issuecomment-1204125004, or unsubscribe https://github.com/notifications/unsubscribe-auth/AADIWFAEMBKTOONTMYJ4MFLVXKHIRANCNFSM55OZFNQQ . You are receiving this because you authored the thread.Message ID: @.***>

Artrois commented 1 year ago

try export GOOGLE_APPLICATION_CREDENTIALS=KEY_PATH then run node

johnfelipe commented 1 year ago

The script uses Google Cloud Translation library. Did you set env properly? To use client libs put path to key.json to env path: PowerShell $env:GOOGLE_APPLICATION_CREDENTIALS="KEY_PATH"

CMD set GOOGLE_APPLICATION_CREDENTIALS=KEY_PATH

root@ubuntu20virgeninwindows:~/Google_Translator_API# set GOOGLE_APPLICATION_CRE                                      DENTIALS=blablablablablablabla

root@ubuntu20virgeninwindows:~/Google_Translator_API# node .\translate_PO_YML.js ./sdg.yml

node:internal/modules/cjs/loader:936
  throw err;
  ^

Error: Cannot find module '/root/Google_Translator_API/.translate_PO_YML.js'
    at Function.Module._resolveFilename (node:internal/modules/cjs/loader:933:15                                      )
    at Function.Module._load (node:internal/modules/cjs/loader:778:27)
    at Function.executeUserEntryPoint [as runMain] (node:internal/modules/run_ma                                      in:77:12)
    at node:internal/main/run_main_module:17:47 {
  code: 'MODULE_NOT_FOUND',
  requireStack: []
}
johnfelipe commented 1 year ago
export GOOGLE_APPLICATION_CREDENTIALS=blablabla

root@ubuntu20virgeninwindows:~/Google_Translator_API# node .\translate_PO_YML.js ./sdg.yml                            node:internal/modules/cjs/loader:936
  throw err;
  ^

Error: Cannot find module '/root/Google_Translator_API/.translate_PO_YML.js'
    at Function.Module._resolveFilename (node:internal/modules/cjs/loader:933:15)
    at Function.Module._load (node:internal/modules/cjs/loader:778:27)
    at Function.executeUserEntryPoint [as runMain] (node:internal/modules/run_main:77:12)
    at node:internal/main/run_main_module:17:47 {
  code: 'MODULE_NOT_FOUND',
  requireStack: []
}

and this one

node translate_PO_YML.js sdg.yml
Wrong parameters given: node translate_PO_YML.js [*.po | *.yml] [target_language]. Supported files: *.po, *.yml. See example:
node .  ranslate_PO_YML.js ./domain.yml en
Artrois commented 1 year ago

there is a problem with your node. The error tells me that node loader fails to load the module did you try to enter: node translate_PO_YML.js sdg.yml Apart from that, your yml file wont be consumed by the script as it is formatted in a different way and not the way the script expects.

johnfelipe commented 1 year ago

wich version of node?

node -v v16.16.0

Artrois commented 1 year ago

try this way: node translate_PO_YML.js sdg.yml en the third parameter is not implemented yet. I planned to propagate target language as third parameter but didnt finish yet.

johnfelipe commented 1 year ago

try this way: node translate_PO_YML.js sdg.yml en the third parameter is not implemented yet. I planned to propagate target language as third parameter but didnt finish yet.

node translate_PO_YML.js sdg.yml en

File exists: sdg.yml, continue counting chars. Detected extension: .yml 1: en: 2: sdg: 3: goals: 4: goal_1: 5: title: "No Poverty" 6: title_in_two_lines: "No\nPoverty" 7: description: "End poverty in all its forms, everywhere." . . .

Total number chars to be translated: 0 Do you want to start translation of YAML? [y/n]:

Artrois commented 1 year ago

yea, the script didnt take the format of your yml.

johnfelipe commented 1 year ago

yea, the script didnt take the format of your yml.

can u give me a section with correct format? then i will convert may be with a regex and then try translate

Artrois commented 1 year ago

here is the major parsing happening: image if you can spare some time take a look at this part and amend accordingly to match your yml format.

johnfelipe commented 1 year ago

yea, the script didnt take the format of your yml.

can u give me a section with correct format? then i will convert may be with a regex and then try translate

is better for me formatting with your app, give me an example using my yaml but converted...

Artrois commented 1 year ago

you would like to translate all lines that contain text in brackets? e.g.: title: "No Poverty" title_in_two_lines: "No\nPoverty" description: "End poverty in all its forms, everywhere." etc. but what about this one: long_description: '

Globally, the number of people living in extreme poverty.......'

johnfelipe commented 1 year ago

this is an example manually:

https://gist.github.com/164b623aadc5c2442005d5342f3c9c6b to https://gist.github.com/95ea691c35b7dacfae23a64c769b4318

Artrois commented 1 year ago

you basically need to translate any text that comes after a colon?

Artrois commented 1 year ago

a proper regex is required and some testing. will look into it later this day

johnfelipe commented 1 year ago

this is an example manually:

https://gist.github.com/164b623aadc5c2442005d5342f3c9c6b to https://gist.github.com/95ea691c35b7dacfae23a64c769b4318

pls use examples and give me a section manually formatted i will create regex and share u for create a branch

johnfelipe commented 1 year ago

what about this regex?

https://regex101.com/r/mjsRjd/4

with that format can i translate with your app?

Artrois commented 1 year ago

I just dropped view lines from your yml into the test field and the given regex shows no match: image

Artrois commented 1 year ago

The brackets are the issue. If we remove the brackets it will match the regex

Artrois commented 1 year ago

If we omit the negative look ahead it will also match brackets: ^\h++[^\s:]++:\h*+\K(["'])(.+)

Artrois commented 1 year ago

Look here: image

Artrois commented 1 year ago

But, the given regex is for PHP and we need for JavaScript. Changing regex flavor shows errors: image

johnfelipe commented 1 year ago

Pls elaborate What do u need?

El jue, 4 de ago. de 2022, 3:23 p. m., Artrois @.***> escribió:

But, the given regex is for PHP and we need for JavaScript. Changing regex flavor shows errors: [image: image] https://user-images.githubusercontent.com/76772066/182945583-36f9b6b4-b940-435b-96ec-44480339d79d.png

— Reply to this email directly, view it on GitHub https://github.com/Artrois/Google_Translator_API/issues/1#issuecomment-1205730493, or unsubscribe https://github.com/notifications/unsubscribe-auth/AADIWFEGAOJJAALFDUOSKHDVXQREZANCNFSM55OZFNQQ . You are receiving this because you authored the thread.Message ID: @.***>

johnfelipe commented 1 year ago

pls share a good yml with all specifics detailed special characters

this is my final yml file https://gist.github.com/95ea691c35b7dacfae23a64c769b4318

but i need to change like...... give me a gist with may be a section with 10 lines correctly

Artrois commented 1 year ago

Just created a regex that takes both formats, with quotes and without quotes: image /(?<=:[ ])"?(.+)"?/g

Artrois commented 1 year ago

will update and test the scripts. We need to use slightly modified regex in order to be able to assemble the translated file without loosing data: ([ ]+(.+))(:[ ]+)"?(.+)"?

johnfelipe commented 1 year ago

Remember yml use double quote when inside of text have special characters and need to put double quotes then can you change your app for commit that?

Artrois commented 1 year ago

I updated the scripts. Use it as follows: node translate_PO_YML.js ..\sdg.yml en es en is the source language and es is the target language. I tested with [(https://gist.github.com/1641f7402fdfc3969e9edb7f08bccc8e)] Remember to amend projectId in translate_strings_with_googleAPI.js I would recommend to make your yml small and dont include much text for the first try. Your full file has lots of characters to translate and might exceed the Translate API limits and might waste your money if the script fails.

johnfelipe commented 1 year ago

one question, for translation in google cloud is API-KEY or service account?

Artrois commented 1 year ago

Im using API-KEY.

Artrois commented 1 year ago

you will need to set env var with the path to the key: export GOOGLE_APPLICATION_CREDENTIALS=KEY_PATH

Artrois commented 1 year ago

the script generates an array of strings from the entire yml and sends in a bunch to google translate API. Recommend to make the yml small and let the script send just few lines for testing. when you are fine with the results, then try the full yml file. But be aware of the API translation limits. Google has translation limits per day and minute as well. With a free tier i think you can translate upto 60000 chars for free

johnfelipe commented 1 year ago

yes im using

export GOOGLE_APPLICATION_CREDENTIALS=AIzaSyCGL-blablbalblalba SNAG-0094

and not working

Do you want to start translation of YAML? [y/n]: y
Translating in 2 secs...
node:fs:2503
      handleErrorFromBinding(ctx);
      ^

Error: The file at AIzaSyCGL-blbalbalblabal does not exist, or it is not a file. ENOENT: no such file or directory, lstat '/root/Google_Translator_API/AIzaSyCGL-blablbalblablabl'
    at Object.realpathSync (node:fs:2503:7)
    at GoogleAuth._getApplicationCredentialsFromFilePath (/root/Google_Translator_API/node_modules/google-auth-library/build/src/auth/googleauth.js:275:27)
    at GoogleAuth._tryGetApplicationCredentialsFromEnvironmentVariable (/root/Google_Translator_API/node_modules/google-auth-library/build/src/auth/googleauth.js:216:25)
    at GoogleAuth.getApplicationDefaultAsync (/root/Google_Translator_API/node_modules/google-auth-library/build/src/auth/googleauth.js:146:24)
    at GoogleAuth.getClient (/root/Google_Translator_API/node_modules/google-auth-library/build/src/auth/googleauth.js:610:28)
    at GrpcClient._getCredentials (/root/Google_Translator_API/node_modules/google-gax/build/src/grpc.js:145:40)
    at GrpcClient.createStub (/root/Google_Translator_API/node_modules/google-gax/build/src/grpc.js:308:34) {
  errno: -2,
  syscall: 'lstat',
  code: 'ENOENT',
  path: '/root/Google_Translator_API/AIzaSyCGL-blablbalblablab'
}
Artrois commented 1 year ago

google API lib cannot find the key. i suggest you put absolute path to the key file: export GOOGLE_APPLICATION_CREDENTIALS=/root/Google_Translator_API/AIzaSyCGL-blablbalblablab then check with ls -a $GOOGLE_APPLICATION_CREDENTIALS

johnfelipe commented 1 year ago

Then that will be json file for service account not api key, right?

El vie, 5 ago 2022 a las 8:41, Artrois @.***>) escribió:

google API lib cannot find the key. i suggest you put absolute path to the key file: export GOOGLE_APPLICATION_CREDENTIALS=/root/Google_Translator_API/AIzaSyCGL-blablbalblablab then check with ls -a $GOOGLE_APPLICATION_CREDENTIALS

— Reply to this email directly, view it on GitHub https://github.com/Artrois/Google_Translator_API/issues/1#issuecomment-1206472354, or unsubscribe https://github.com/notifications/unsubscribe-auth/AADIWFCERTZWBD5NK7QWZNTVXUKXZANCNFSM55OZFNQQ . You are receiving this because you authored the thread.Message ID: @.***>

Artrois commented 1 year ago

Yea, my bad! I think you are right. I'm also using JSON file but I don't remember if it's for service account.

johnfelipe commented 1 year ago
export GOOGLE_APPLICATION_CREDENTIALS=traducciones-352415-79659df1c6f1.json

ls -a $GOOGLE_APPLICATION_CREDENTIALS
traducciones-352415-79659df1c6f1.json

then still happend error here:

Do you want to start translation of YAML? [y/n]: y
Translating in 2 secs...
/root/Google_Translator_API/node_modules/@grpc/grpc-js/build/src/call.js:31
    return Object.assign(new Error(message), status);
                         ^

Error: 3 INVALID_ARGUMENT: Text is too long.
    at Object.callErrorFromStatus (/root/Google_Translator_API/node_modules/@grpc/grpc-js/build/src/call.js:31:26)
    at Object.onReceiveStatus (/root/Google_Translator_API/node_modules/@grpc/grpc-js/build/src/client.js:189:52)
    at Object.onReceiveStatus (/root/Google_Translator_API/node_modules/@grpc/grpc-js/build/src/client-interceptors.js:365:141)
    at Object.onReceiveStatus (/root/Google_Translator_API/node_modules/@grpc/grpc-js/build/src/client-interceptors.js:328:181)
    at /root/Google_Translator_API/node_modules/@grpc/grpc-js/build/src/call-stream.js:187:78
    at processTicksAndRejections (node:internal/process/task_queues:78:11) {
  code: 3,
  details: 'Text is too long.',
  metadata: Metadata {
    internalRepr: Map(3) {
      'grpc-server-stats-bin' => [
        Buffer(10) [Uint8Array] [
          0, 0, 220, 24, 73,
          7, 0,   0,  0,  0
        ]
      ],
      'google.rpc.badrequest-bin' => [
        Buffer(88) [Uint8Array] [
           10,  86,  10,   8,  99, 111, 110, 116, 101, 110, 116, 115,
           18,  74,  84, 104, 101,  32, 116, 111, 116,  97, 108,  32,
           99, 111, 100, 101, 112, 111, 105, 110, 116, 115,  32, 105,
          110,  32, 116, 104, 101,  32, 114, 101, 113, 117, 101, 115,
          116,  32, 109, 117, 115, 116,  32,  98, 101,  32, 108, 101,
          115, 115,  32, 116, 104,  97, 110,  32,  51,  48,  55,  50,
           48,  44,  32,  97,  99, 116, 117,  97, 108,  58,  32,  55,
           53,  52,  54,  56
        ]
      ],
      'grpc-status-details-bin' => [
        Buffer(157) [Uint8Array] [
            8,   3,  18,  17,  84, 101, 120, 116,  32, 105, 115,  32,
          116, 111, 111,  32, 108, 111, 110, 103,  46,  26, 133,   1,
           10,  41, 116, 121, 112, 101,  46, 103, 111, 111, 103, 108,
          101,  97, 112, 105, 115,  46,  99, 111, 109,  47, 103, 111,
          111, 103, 108, 101,  46, 114, 112,  99,  46,  66,  97, 100,
           82, 101, 113, 117, 101, 115, 116,  18,  88,  10,  86,  10,
            8,  99, 111, 110, 116, 101, 110, 116, 115,  18,  74,  84,
          104, 101,  32, 116, 111, 116,  97, 108,  32,  99, 111, 100,
          101, 112, 111, 105,
          ... 57 more items
        ]
      ]
    },
    options: {}
  },
  statusDetails: [
    BadRequest {
      fieldViolations: [
        FieldViolation {
          field: 'contents',
          description: 'The total codepoints in the request must be less than 30720, actual: 75468'
        }
      ]
    }
  ]
}
Artrois commented 1 year ago

The file is too big for google API. The error tells you this: The total codepoints in the request must be less than 30720, actual: 75468' You need to check the limits set for the API, maybe it is possible to increase for more than 75000 per request.

Artrois commented 1 year ago

In the meantime I suggest you shorten your yml file to few lines for translation and try to translation. This will give you the confidence that your yml is getting consumed and the script works with your Google credentials.

Artrois commented 1 year ago

@johnfelipe did it work?

johnfelipe commented 1 year ago

in your not

Artrois commented 1 year ago

As mentioned earlier, google API has limits per request and your request exceeded the limit of allowed characters. Check your API settings and increase the limits