Doffin validation API returns false whereas Validator website and TED validation API successfully validate the same file

[aso-ivalua]

Hello, Yesterday I noticed that all my test notices were failing using the doffin validation API, but they are successfully validated by the validation website (https://anskaffelser.dev/service/validator/) and by the TED api (after transformation using nor-to-eforms.xslt). I even tried with a file that was previously validated by the API but it now just returns 'success: false'.

Please find the file in attachment. Thank you pin_only.txt

[erik-itland]

I will take a look at it.

[erik-itland]

It seems the file has been submitted to TED from our side. I write seems because what I have is an svrl-report which isn't supposed to be there unless we have sent the file, but I haven't traced it out yet.

If it is sent to TED you should have received a mail with a report to that email address. Could you paste the body of that email here if you have received it?

[aso-ivalua]

You are right, this notice ID was submitted previously and I am aware that the submission API is supposed to fail. However, the validation API should still give success because the format is correct. I also have other non submitted examples of correct notices that pass the online validator but not the validations api.

Here is the last email for this notice :

Your notice 6e2ad497-1da6-4dae-a63c-39ffcf19a8b9-01 has been published, you can see it in your account with the status "Published".

Thank you

Le mar. 14 nov. 2023 à 12:07, erik-itland @.***> a écrit :

It seems the file has been submitted to TED from our side. I write seems because what I have is an svrl-report which isn't supposed to be there unless we have sent the file, but I haven't traced it out yet.

If it is sent to TED you should have received a mail with a report to that email address. Could you paste the body of that email here if you have received it?

— Reply to this email directly, view it on GitHub https://github.com/anskaffelser/eforms-sdk-nor/issues/46#issuecomment-1810000154, or unsubscribe https://github.com/notifications/unsubscribe-auth/BD6O3VALRAFFERDRGAVOYILYENGINAVCNFSM6AAAAAA7KRK5IKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQMJQGAYDAMJVGQ . You are receiving this because you authored the thread.Message ID: @.***>

[aso-ivalua]

Here is an example of a file not yet submitted that passes the validation website but not the validation API.

Validator URL: https://anskaffelser.dev/service/validator/224dec42-d413-461f-bf87-2b7a07fec76e File: Dof_Notice20231114125730.txt

[erik-itland]

Hi again!

We just discovered one interesting detail:

If we change the filename to "pin_only.xml" it validates fine.

We are already looking into the code to figure out why this happens, but until then, as a workaround, can you try using the extension ".xml" ?

[erik-itland]

Do you by any chance use Windows notepad to adjust the files? I'm guessing here because of the .txt file extension and the fact that the file includes a BOM even if it is otherwise a UTF-8-file and that is something older versions of Notepad in Windows is known to do.

I am still looking into how we can accommodate your files. In my opinion, if the files validate in TED or the anskaffelser.dev validator they should validate in our code as well.

PS: I just realized one thing about the xml/txt thing: It wasn't related to the changing of the file extension but to the fact that we had opened the file in our IDE and saved a copy with the file extension .xml which meant that the BOM was removed.

[aso-ivalua]

Hello,

I renamed my original xml file to .txt only because github doesn’t allow uploading an xml file. You can get the xml file from my last anskaffelser link by clicking on download source.

Le mer. 15 nov. 2023 à 10:20, erik-itland @.***> a écrit :

Do you by any chance use Windows notepad to adjust the files? I'm guessing here because of the .txt file extension and the fact that the file includes a BOM even if it is otherwise a UTF-8-file and that is something older versions of Notepad in Windows is known to do.

I am still looking into how we can accommodate your files. In my opinion, if the files validate in TED or the anskaffelser.dev validator they should validate in our code as well.

PS: I just realized one thing about the xml/txt thing: It wasn't related to the changing of the file extension but to the fact that we had opened the file in our IDE and saved a copy with the file extension .xml which meant that the BOM was removed.

— Reply to this email directly, view it on GitHub https://github.com/anskaffelser/eforms-sdk-nor/issues/46#issuecomment-1812079339, or unsubscribe https://github.com/notifications/unsubscribe-auth/BD6O3VE5222ZJKUAWYFM6K3YESCMRAVCNFSM6AAAAAA7KRK5IKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQMJSGA3TSMZTHE . You are receiving this because you authored the thread.Message ID: @.***>

[erik-itland]

I see.

I included it in the PS on my last reply but I guess our messages flew past each other.

Meanwhile we have localized the problem to a very new part of our code that deals with the fact that sometimes Azure sends the files with "windows-1252" encoding instead of UTF-8 (seems Azure har some faulty heuristic in the somewhere to detect file encoding).

We are working on a code change to fix this and will add a new integration test to make sure it stays fixed as well.

For now, if you send the file without a BOM it should work.

I also notice that your file does not include an xml declaration on the top. This should not be a problem for our code, but it is included in xml 1.1 which was released in 2006.

[erik-itland]

Hi again!

I have now finished a PR to fix the issues you were seeing and it has also been appproved, merged and should now be live in the testing environment.

Please let me know when you have had time to test it and feel free to send a few more files as well if the first file works.

[aso-ivalua]

It is working now indeed ! thank you very much

[erik-itland]

No problem and thanks for closing the issue! Have a nice day!

[kjorlaug]

Hi @aso-ivalua - I see that you are using version 1.6 of the SDK (we see that in the CustomizationId). Would you consider to start using 1.7 or 1.8 instead? TED will stop supporting 1.6 by the end of March next year. Supporting versions prior to 1.7 makes maintaining the national forms (N*) more complex for us, so if possible we would like to stop supporting it soon.

--

Rune

[aso-ivalua]

Hi @kjorlaug , It seems we are encountering the same issue again with notices in BOM format failing on 1.6. When using the same file in 1.7 the validation is successful. We are in the process of switching to either 1.7 or 1.8 right now, we are able to successfully validate basic notices where only the required information is filled. However, we do not use your sdk so all rules are manually implemented in our product. In order to fully switch to 1.7 in production we need to make sure that any new rules or other changes are covered. Is there any document that can help identify the changes from 1.6 to 1.7 (or 1.8). Thank you

[erik-itland]

FWIW I'll just mention that in the dev environment I have already removed the 1.6 SDK this week so the reason why 1.6 doesn't work anymore is because there is nothing there to validate 1.6 files with.

[aso-ivalua]

Then we have to switch to 1.7 right away, we will implement any rules as we encounter them, but we won't be able to make sure that we cover all scenarios. Again, if you have any summary of changes between 1.6 and 1.7, it will be of huge help.