Several tries to create data structures implicitly

HolQue commented 8 months ago

[1]

In second line of following JSON code

"testdict" : {"A" : 1},
${testdict.B}['C'] : 2

I use a mixture of dotdict syntax and standard syntax to create a data structure implicitly.

This works properly! Outcome: {'testdict': {'A': 1, 'B': {'C': 2}}}

[2]

Now I use a parameter for the name of a dictionary key:

"testdict" : {"A" : 1},
"name" : "C",
${testdict.B}['${name}'] : 2

Outcome:

'Could not set variable 'testdict.B['C']' with value '2'! Reason: 'dict' object has no attribute 'B'

Clarification wanted: Is this a valid and confirmed syntax? If not, the error message should be reduced to something like 'invalid syntax'. Because the current error message is too much technical and does not help the users.

[3]

Alternative try:

"testdict" : {"A" : 1},
"name" : "C",
${testdict.B.${name}} : 2

Outcome:

{'${testdict.B.${name}}': 2, 'name': 'C', 'testdict': {'A': 1}}

The parameter name is not resolved properly.

If the syntax is valid, the parameter name should be resolved properly, otherwise 'invalid syntax' error message.

[4]

Alternative try:

"testdict" : {"A" : 1},
"name" : "C",
${testdict}['${name}']['${name}'] : 2

This works properly!

{'name': 'C', 'testdict': {'A': 1, 'C': {'C': 2}}}

[5]

Alternative try:

"testdict" : {"A" : 1},
"name" : "C",
${testdict}.${name}.${name} : 2

Outcome:

Invalid nested parameter format: ${testdict}.${name}.${name} - The double quotes are missing!!!

I have no concrete idea about what can be the expectation here, but my feeling is that this really is invalid.

I would assume that the closing bracket after testdict has to be placed at the end of the expression (next example).

[6]

Alternative try:

"testdict" : {"A" : 1},
"name" : "C",
${testdict.${name}.${name}} : 2

Outcome:

Invalid nested parameter format: ${testdict.${name}.${name}} - The double quotes are missing!!!'

Now I am astonished. My expectation would be that the dictionary {'A': 1, 'C': {'C': 2}} is created.

Interesting: This is similar to [3] (but in [3] only one parameter is used, the second key is hard coded, but this should not make a difference).

test-fullautomation commented 8 months ago

Hi Holger,

puhhh.... Let us discuss the use-cases for the 0.10.0 release.

In principle we agree that we don't mix both syntaxes in one expression, but all described uses-cases should work from my point of view, too because I would consider each $-expression as an own entity.

Thank you, Thomas

HolQue commented 8 months ago

Hi Thomas,

if the described uses cases shall work, this issue is a bug, not an enhancement.

namsonx commented 6 months ago

Hello Holger,

I'm still working on this ticket, user cases [1] to [4] are working, [5] and [6] in progress. I will push new commit to stabi branch when completed all user cases

Thank you, Son

namsonx commented 6 months ago

Hello Holger,

Now, the user cases [5] and [6] confuse me because:

In user case [5], ${testdict}.${name}.${name} : 2 is invalid syntax because the . is located outside of ${testdict}
in user case [6], if the ${testdict.${name}.${name}} : 2 is valid, but we also have ${name}.${name} where the . also is located outside of ${name}. And in this case, we will consider the . is invalid format of ${name} or a sub-element of testdict.${name}?

Thank you, Son

HolQue commented 6 months ago

Hello Son,

you're right. Example [6] is also invalid (in the same way as [5]). I overlooked the inner structure.

If my understanding is right now, this should be valid:

${testdict.${name.${name}}}

namsonx commented 6 months ago

Hello Holger,

The use cases in this ticket are implemented in latest commits on stabi branch.

Thank you, Son

HolQue commented 6 months ago

13.12.2023 / Addendum

Hi Son,

I am sorry, but I completely have lost the overview.

A dict and a string:

"testdict" : {"A" : 1},
"name" : "C",

Another key for the dict:

${testdict}['${name}'] : 2
${testdict.${name}} : 3,

Booth work.

But how to add another sub key?

(1) I tried:

${testdict}['${name}']['${name}'] : 4

With result:

Could not set variable 'testdict['C']['C']' with value '4'! Reason: 'int' object does not support item assignment'

What does this mean?

Addendum 14.12.2023: Fixed in latest stabi branch!

(2) Just to see what happened, I removed the quotes (but I know I should not do that):

${testdict}[${name}][${name}] : 4

Result:

${testdict}[${name}][${name}] : 4

But the JsonPreprocessor should not return something that contain unresolved dollar operator expressions.

(3) I tried:

${testdict.${name}.${name}} : 4

Result:

Invalid parameter format: ${testdict.${name}.${name}}

But what exactly is invalid?

(4) I tried:

${testdict.${name.${name}}} : 4

Result:

The variable '${name.C}' is not available!

This error is OK, we have already clarified this.

Alternatively I tried:

"testdict2" : {"B" : 2},
${testdict.${testdict2.${name}}} : 4

Result:

The variable '${testdict2.C}' is not available!

But why not?

Addendum 14.12.2023: My misunderstanding; error is correct!

Alternatively I tried:

"testdict2" : {"B" : 2},
${testdict.${testdict2}}['${name}'] : 4

!!! This makes the JsonPreprocessor freeze !!!

(5) I tried:

${testdict.${name}}['${name}'] : 5

Result:

'NoneType' object has no attribute 'group'

What does this mean?

Addendum 14.12.2023: another combination

(6) I tried:

"testdict" : {"A" : "B", "B" : "C", "C" : 1},
"intval_1" : ${testdict}[${testdict}[${testdict}['A']]]

Result: intval_1 is 1, like expected.

I tried:

"intval_2" : ${testdict.${testdict.B}}

Result:

'Invalid syntax! One or more than one opened or closed curly bracket is missing in expression '${testdict.${testdict.B}}'

Why this? There is no bracket mismatch in this expression.

I tried:

"intval_3" : ${testdict.${testdict.${testdict.A}}}

Result:

'Invalid nested parameter format: ${testdict.${testdict.${testdict.A}}} - The double quotes are missing!!!'

Now the the error message is completely different. It is astonishing that this depends on the number of subkeys.

Expected is still integer 1.

Can we please have a clarification here? I have no idea any more what is correct and what is incorrect.

test-fullautomation commented 6 months ago

Hi Son, I agreed with Holger that we block implicit creation of data structures based on parameters. This will be an accepted limitation in order to reduce the complexity. If we have later a use-case we can implement this.

Implicit creation shall have the need to hard code the name.

Please create a speaking error message if a referenced key is not existing.

Thank you, Thomas

HolQue commented 6 months ago

Let's see if I got the consequences.

This is allowed because the names of all not existing keys are hard coded:

"testdict" : {"A" : 1},
${testdict.B}['C'] : 2

The following three versions are currently working fine! But the key names are not hard coded, they are defined by parameters.

If my understanding is correct, this shall now not be allowed any more. Right?

"testdict" : {"A" : 1},
"name" : "C",
${testdict.B}['${name}'] : 2

"testdict" : {"A" : 1},
"name" : "C",
${testdict.B.${name}} : 2

"testdict" : {"A" : 1},
"name" : "C",
${testdict}['${name}']['${name}'] : 2

Then this would be the required way of computation:

Parameter ${name} not existing => Error: not existing parameter
Parameter ${name}exists, but a key with this name does not exist in testdict => Error: Implicit creation of data structures based on parameters not allowed. Key names must be hard coded in this case.

Am I right?

namsonx commented 5 months ago

(1) I tried:

${testdict}['${name}']['${name}'] : 4

With result:

Could not set variable 'testdict['C']['C']' with value '4'! Reason: 'int' object does not support item assignment'

What does this mean?

Addendum 14.12.2023: Fixed in latest stabi branch!

Hello Holger,

Regarding to the case (1) you mentioned:

(1)
I tried:

${testdict}['${name}']['${name}'] : 4

With result:

Could not set variable 'testdict['C']['C']' with value '4'! Reason: 'int' object does not support item assignment'

What does this mean?

Addendum 14.12.2023: Fixed in latest stabi branch!

I guess this error message log out due to you set ${testdict.${name}} : 3, before, then it could not add sub element for int datatype. I reproduced this case and it worked fine from my site.

Nevertheless, as Thomas' comment above we will disable implicit creation of data structures based on parameters. I will disable this feature, then we just allow for example ${testdict}['C']['C'] or ${testdict.C.C}

Thank You, Son

namsonx commented 5 months ago

Hello Thomas, Hello Holger,

I temporary disabled implicit creation of data structures based on nested parameters, and created new pull-request #181 for it.

Thank you, Son

HolQue commented 5 months ago

Because of the disabled implicit creation of data structures, most of the use cases above are invalid now.

Therefore a follow-up investigation is necessary.

First outcome:

This code

"name" : "C",
"testdict2" : {"B" : 2},
${testdict.${testdict2}}['${name}'] : 4

still causes a crash of the JsonPreprocessor. But even in case of the code is invalid, the JsonPreprocessor should not crash.

HolQue commented 5 months ago

The (invalid) code

"testdict" : {"A" : 1},
"name" : "C",
${testdict}['${name}']['${name}'] : 4

causes the following error message:

Error: '
The implicit creation of data structures based on nested parameter does not enable yet.
New parameter 'testdict['C']['C']' could not be created by the expression '${testdict}['${name}']['${name}']''!

(1) When you wrap the error message in single quotes, then please avoid line breaks directly after the first quote. This looks strange.

(2) Please take care of your wording: "does not enable yet". Better: "... is not supported."

HolQue commented 5 months ago

How to handle:

"testdict" : {"A" : 1},
"name" : "C",
${testdict}[${name}][${name}] : 4

Result:

${testdict}[${name}][${name}] with value 4.

But in my opinion the returned value from JsonPreprocessor should not contain unresolved dollar operator expressions.

HolQue commented 5 months ago

This is also a not supported try to define a data structure implicitly based on parameters:

"testdict" : {"A" : 1},
"name" : "C",
${testdict.${name}}['${name}'] : 5

Result:

Error: ''NoneType' object has no attribute 'group''!

This is not helpful for users. Also in this case expected is an error message telling that implicit creation of data structures based on parameters is not supported.

namsonx commented 5 months ago

Hello Holger,

Thank you for your findings! I resolved all 4 issues you mentioned in 4 comments above via new commit 34a35aa697fc2

Thank you, Son

HolQue commented 5 months ago

Retest successful. Issue can be closed.

namsonx commented 4 months ago

merged to develop branch

test-fullautomation commented 4 months ago

solved with version 0.10.0

test-fullautomation / python-jsonpreprocessor

Several tries to create data structures implicitly #153