Unrecognized token "]" - Githubissues

jamesvl commented 1 year ago

I'm attempting to use scrumdog for an initial import from Jira but the executing fails pretty quickly with

2023-01-21 09:57:42 - SQLite: Create table 'dx_fields'
2023-01-21 09:57:42 - SQLite: Create table 'dx_issues'
2023-01-21 09:57:42 - Jira: Get Issue Fields
ERROR
unrecognized token: "]"
db: unable to execute sql
db: exiting db ...

I'm not sure if the ] is from a tag or field in my Jira instance or bug in the SD parsing - is there a way I can provide any further info to help debug this?

whoek commented 1 year ago

@jamesvl Thanks for logging the issue.

Look like the error message is coming back from Jira itself. are you using Windows, Max or Linux - and which version of OS? I'll add more error messages and release a new version shortly (few days)

Thanks

jamesvl commented 1 year ago

Sure, I'm on Linux (Fedora 37), using the latest scrumdog version I believe:

scrumdog 0.4  build 558 (64-bit)
Release-Date: 2022-07-27

whoek commented 1 year ago

@jamesvl -- I have made some changes to scrumdog so that it does not rely on curl any more. The curl dependency did not work for all users. The latest downloads are available from here. Thanks

jamesvl commented 1 year ago

I tried the latest version (scrumdog 0.5 build 611 (64-bit)), but unfortunately it's still giving me the same error....

whoek commented 1 year ago

@jamesvl --

1 - JIRA CLOUD Scrumdog pull issues from JIRA Cloud rather than JIRA Server. Lets know if you using JIRA Cloud.

2 - LOG FILES When running scrumdog, it will save a few status files in a log folder (in the same directory as where the executable is stored). In this folder -- you should see following files:

File ending with fields_code.txt -- the return code should be 200
File ending with fields_header.txt -- server should be Server: AtlassianEdge
File ending with fields_body.json should contain return data (body).

Have a look in these files for a clue.

3 - NEW MINOR VERSION I did create a new version -- although not sure if it will solve all of this.

Thanks again for testing scrumdog

jamesvl commented 1 year ago

Still getting the same error. Could it be because some of our custom field names have brackets in them, and somehow this is tripping up your SQL escaping?

All _code.txt files returned 200, all _header files had the server name you indicated. I'm attaching the last JSON body that was logged, in case it help you at all. (It's only the field name details, so no actual data from my Jira tickets. Also, I pretty-formatted the JSON for readability; the original came back with everything on one line.)

fields.zip

whoek commented 1 year ago

Hello @jamesvl

Many Thanks for looking at this again and taking the trouble to provide the Jira fields json file.

The error message is Sqlite complaining about a ] character in either a table or field name. I did some tests using your json file and was unable to replicate the error. This means that the Jira field names in your Jira setup is not the issue -- which is a good start!

I was able to replicate the error if I add a ] in the setup file.

Steps followed to replicate error

I first created a default setup file by running scrumdog -j. This will create a file called sample.jql.

In sample.jql -- you would have updated the first 5 values as per your Jira credentials and project details.

[server]     https://yourcompany.atlassian.net/
[email]      name@yourcompany.com
[api_token]    xxxxYOURAPITOKENxxxxxxxxx
[jql]              project = zz
[db_filename]   jira.db
[db_table_prefix]  zz

[fields]

assignee   assignee.displayName
creator    creator.displayName

Now -- if I add a ] to either the [db_table_prefix] or fields, I get the same error. Example

[server]     https://yourcompany.atlassian.net/
[email]      name@yourcompany.com
[api_token]    xxxxYOURAPITOKENxxxxxxxxx
[jql]              project = zz
[db_filename]   jira.db
[db_table_prefix]  zz]

[fields]

assignee   assignee.displayName
creator    creator.displayName

Note the ] after the zz in [db_table_prefix] token. I suspect you might have added a ] character in your setup file.

Some notes on handling custom fields

Best is -- start off by not defining your custom fields in the setup file. Just get it to pull all the info from Jira first. You should then see a list of ALL your fields (including custom ones) in the fields database table and it will also be in your issues table.

The purpose of the [fields] tag in setup is to either rename the fieldname or, more importantly pick up the right value from the json

every row contains 2 strings
the first one is what the field-name will be in the sqlite database
the second string is, where in the return json to get the value

Here is a simple example of a custom field in my project. In this case the program automatically adds the custom field and give it a fieldname based on best guess, which can be changed in [fields] tag.

issues-table

Then pick up the relevant JSON from issues table (in Sqlite).

fields

I then copy / paste the json in a json viewer -- there are many online ones -- I used https://jsonviewer.stack.hu/

custom

So in this case -- I can add following to my [fields] section

country_list  customfield_10031

which will change the fieldname in the issues table country-list

Back to your initial error message

Do you have any ] characters in your setup file?
For now I would suggest you remove them and use simple string in your [db_table_prefix] tag - just to get it all working.

Also -- it would be great to understand your use case so that I can change the program to be more robust or provide assistance in making it work for you.

Thanks again Willem

Above is all based on v0.51 build 667 of April 2, 2023 Download from here -- https://github.com/whoek/scrumdog-binaries

whoek / scrumdog

Unrecognized token "]" #2