With newer compiler -march options may cause issues between DUT and Sail

[InspireSemi]

Sometime after gcc 10.2, the -march option has changed. For 10.2 rv64i also carried with it _zicsr. This is no longer the case with gcc 12 and higher.

If a DUT outside of the test code (model.h) reads/writes a csr it will fail at compile under gcc 12. The -march is passed into the python prog/script that generates the Makefile. For example under I extension it uses rv64i.

If the DUT changes this in the python code to be hardcoded to some other value, this may and can modify the DUT generated code vs the Sail generated code resulting in a signature diff between the two.

I ran into this as I hardcoded the python to be -march=rv64imafdc_zicsr_zifencei. The jalr and jal opcodes under I failed. I could not figure out why as both codes if you followed the code execution came up with correct signatures for themselves but different for DUT and Sail.

I then noticed that the DUT code had compressed opcodes in it the Sail code did not. This was causing address differences that would make the signatures calulated based on the addresses in the code different.

[allenjbaum]

Were the compressed ops in the tests themselves or in the trap handler? We were pretty careful in the trap handler to ensure that it used no compressed ops (except as c.nop as padding, IIRC)

On Mon, Jul 24, 2023 at 9:05 PM InspireSemi @.***> wrote:

Sometime after gcc 10.2, the -march option has changed. For 10.2 rv64i also carried with it _zicsr. This is no longer the case with gcc 12 and higher.

If a DUT outside of the test code (model.h) reads/writes a csr it will fail at compile under gcc 12. The -march is passed into the python prog/script that generates the Makefile. For example under I extension it uses rv64i.

If the DUT changes this in the python code to be hardcoded to some other value, this may and can modify the DUT generated code vs the Sail generated code resulting in a signature diff between the two.

I ran into this as I hardcoded the python to be -march=rv64imafdc_zicsr_zifencei. The jalr and jal opcodes under I failed. I could not figure out why as both codes if you followed the code execution came up with correct signatures for themselves but different for DUT and Sail.

I then noticed that the DUT code had compressed opcodes in it the Sail code did not. This was causing address differences that would make the signatures calulated based on the addresses in the code different.

— Reply to this email directly, view it on GitHub https://github.com/riscv-non-isa/riscv-arch-test/issues/369, or unsubscribe https://github.com/notifications/unsubscribe-auth/AHPXVJVROWV5XSVC3RULADLXR5AXNANCNFSM6AAAAAA2WOS53E . You are receiving this because you are subscribed to this thread.Message ID: @.***>

[InspireSemi]

It is not how careful you are the, gcc optimized the code to use compressed because I told It that it had Compressed available (set the march to rv64imafdc_zcsr_zfencei) while the sail model got just the rv64i for its compile.

I saw in the two objdumps one from out DUT and one from Sail that our code had compressed opcodes in it sail did not. This was in the test case code. I reverted my change back so the python code was again using the passed in march parameter. Made sure I was compiling with gcc10.2 and magic signatures matched.

What this caused was the DUT to have one signature and Sail to have another signature. This was because the signatures are dependent on the code having the same opcodes. As soon as one side has a 16 bit opcode (for example nop) it throws the signature calculation off.

The issue that started this was I was looking to move internally to gcc12. Tried to run riscof under this. It was failing to compile because it said I had a csr opcode and no support for it??? Found out the under gcc10.2 rv64i implied _zcsr for march. This was changed under gcc 12.x and rv64i is just I now. I made the change above to get it to run under gcc 12 and hardcoded the march in the python code to cover all cases.

This then led to this bug in mismatched signatures.

Marc

Marc Karasek Principal Software Engineer M: 678.770.3788

[A close up of a sign Description automatically generated] www.inspiresemi.comhttp://www.cryptocoretech.com/

THIS MESSAGE IS INTENDED ONLY FOR THE USE OF THE INDIVIDUAL OR ENTITY TO WHICH IT IS ADDRESSED AND MAY CONTAIN INFORMATION THAT IS PRIVILEGED, CONFIDENTIAL AND/OR EXEMPT FROM DISCLOSURE UNDER APPLICABLE LAW. If the reader of this message is not the intended recipient or agent responsible for delivering the message to the intended recipient, then you are hereby notified that any dissemination or copying of this communication is strictly prohibited. If you have received this electronic transmission in error, please delete it from your system without copying it and notify the sender by reply e-mail so that our address record can be corrected. Thank you.

From: Allen Baum @.> Sent: Tuesday, July 25, 2023 1:37 PM To: riscv-non-isa/riscv-arch-test @.> Cc: Marc Karasek @.>; Author @.> Subject: Re: [riscv-non-isa/riscv-arch-test] With newer compiler -march options may cause issues between DUT and Sail (Issue #369)

Were the compressed ops in the tests themselves or in the trap handler? We were pretty careful in the trap handler to ensure that it used no compressed ops (except as c.nop as padding, IIRC)

On Mon, Jul 24, 2023 at 9:05 PM InspireSemi @.<mailto:@.>> wrote:

Sometime after gcc 10.2, the -march option has changed. For 10.2 rv64i also carried with it _zicsr. This is no longer the case with gcc 12 and higher.

If a DUT outside of the test code (model.h) reads/writes a csr it will fail at compile under gcc 12. The -march is passed into the python prog/script that generates the Makefile. For example under I extension it uses rv64i.

If the DUT changes this in the python code to be hardcoded to some other value, this may and can modify the DUT generated code vs the Sail generated code resulting in a signature diff between the two.

I ran into this as I hardcoded the python to be -march=rv64imafdc_zicsr_zifencei. The jalr and jal opcodes under I failed. I could not figure out why as both codes if you followed the code execution came up with correct signatures for themselves but different for DUT and Sail.

I then noticed that the DUT code had compressed opcodes in it the Sail code did not. This was causing address differences that would make the signatures calulated based on the addresses in the code different.

— Reply to this email directly, view it on GitHub https://url.avanan.click/v2/___https://github.com/riscv-non-isa/riscv-arch-test/issues/369<https://url.avanan.click/v2/___https:/github.com/riscv-non-isa/riscv-arch-test/issues/369>.YXAzOmluc3BpcmVzZW1pOmE6bzoyMzA4ZjNlODM1Nzg3NjY1YjkwYWVlNzlkMjc2N2JkMDo2OjFmNjk6NGExODUyYWE0MjZlYzBiMjA0NzhjNTY5MjI4M2ZiMmQ4MDAzZDM5Y2RhYjFiMTRiNDE1YjAzNDVmODU2ODM0MDp0OlQ;, or unsubscribe <https://url.avanan.click/v2/https://github.com/notifications/unsubscribe-auth/AHPXVJVROWV5XSVC3RULADLXR5AXNANCNFSM6AAAAAA2WOS53Ehttps://url.avanan.click/v2/___https:/github.com/notifications/unsubscribe-auth/AHPXVJVROWV5XSVC3RULADLXR5AXNANCNFSM6AAAAAA2WOS53E>___.YXAzOmluc3BpcmVzZW1pOmE6bzoyMzA4ZjNlODM1Nzg3NjY1YjkwYWVlNzlkMjc2N2JkMDo2OmFjZTk6OTVhZjY5ZGRhNTdiZjUyNTk5MDkzMjdkZWZjYjE1MGI3NTQ0NTQ2YjA0YTY4MjFlOTQ4MGNlMTAyNTIwM2U1YTp0OlQ; . You are receiving this because you are subscribed to this thread.Message ID: @.<mailto:@.>>

— Reply to this email directly, view it on GitHubhttps://url.avanan.click/v2/___https:/github.com/riscv-non-isa/riscv-arch-test/issues/369%23issuecomment-1650258617___.YXAzOmluc3BpcmVzZW1pOmE6bzoyMzA4ZjNlODM1Nzg3NjY1YjkwYWVlNzlkMjc2N2JkMDo2OmIzYzk6NWM5ZWU0ZjkzMTg1NTFhOWM5N2UwZmU1ZTQ2YmI4NzY3MjYyZWIwNDk1NTI2OTQzZWQzOTdkOGJlMGUwZTdmYTpoOlQ, or unsubscribehttps://url.avanan.click/v2/___https:/github.com/notifications/unsubscribe-auth/AR3S6Y5RWLGIZREBL5VLUNTXR774LANCNFSM6AAAAAA2WOS53E___.YXAzOmluc3BpcmVzZW1pOmE6bzoyMzA4ZjNlODM1Nzg3NjY1YjkwYWVlNzlkMjc2N2JkMDo2OmE3MTk6MWRmZWNjNDJiOWZiYTU5MzY5OWQyZjZkM2IwMzJlODdjMDk1OWMyYzA5YThmZWE5NTdkODI0OTlmMGJjZjY0NjpoOlQ. You are receiving this because you authored the thread.Message ID: @.**@.>>

[allenjbaum]

GCC shouldn't be able to use compressed ops in the trap handler because it specifically has .option norvc everywhere (except for that one exception). I am confused if this is a bug in GCC, in our tests, in riscof, in Sail, or in your test configuration? If you pass the C extension in -march string, the tools should allow compressed ops to be generated - including for Sail. Why was Sail getting a different compile parameters than your DUT? Were you using GCC12 for one of them and GCC10 for the other?

On Tue, Jul 25, 2023 at 2:45 PM InspireSemi @.***> wrote:

It is not how careful you are the, gcc optimized the code to use compressed because I told It that it had Compressed available (set the march to rv64imafdc_zcsr_zfencei) while the sail model got just the rv64i for its compile.

I saw in the two objdumps one from out DUT and one from Sail that our code had compressed opcodes in it sail did not. This was in the test case code. I reverted my change back so the python code was again using the passed in march parameter. Made sure I was compiling with gcc10.2 and magic signatures matched.

What this caused was the DUT to have one signature and Sail to have another signature. This was because the signatures are dependent on the code having the same opcodes. As soon as one side has a 16 bit opcode (for example nop) it throws the signature calculation off.

The issue that started this was I was looking to move internally to gcc12. Tried to run riscof under this. It was failing to compile because it said I had a csr opcode and no support for it??? Found out the under gcc10.2 rv64i implied _zcsr for march. This was changed under gcc 12.x and rv64i is just I now. I made the change above to get it to run under gcc 12 and hardcoded the march in the python code to cover all cases.

This then led to this bug in mismatched signatures.

Marc

Marc Karasek Principal Software Engineer M: 678.770.3788

[A close up of a sign Description automatically generated] www.inspiresemi.comhttp://www.cryptocoretech.com/

THIS MESSAGE IS INTENDED ONLY FOR THE USE OF THE INDIVIDUAL OR ENTITY TO WHICH IT IS ADDRESSED AND MAY CONTAIN INFORMATION THAT IS PRIVILEGED, CONFIDENTIAL AND/OR EXEMPT FROM DISCLOSURE UNDER APPLICABLE LAW. If the reader of this message is not the intended recipient or agent responsible for delivering the message to the intended recipient, then you are hereby notified that any dissemination or copying of this communication is strictly prohibited. If you have received this electronic transmission in error, please delete it from your system without copying it and notify the sender by reply e-mail so that our address record can be corrected. Thank you.

From: Allen Baum @.> Sent: Tuesday, July 25, 2023 1:37 PM To: riscv-non-isa/riscv-arch-test @.> Cc: Marc Karasek @.>; Author @.> Subject: Re: [riscv-non-isa/riscv-arch-test] With newer compiler -march options may cause issues between DUT and Sail (Issue #369)

Were the compressed ops in the tests themselves or in the trap handler? We were pretty careful in the trap handler to ensure that it used no compressed ops (except as c.nop as padding, IIRC)

On Mon, Jul 24, 2023 at 9:05 PM InspireSemi @.<mailto:@.>>

wrote:

Sometime after gcc 10.2, the -march option has changed. For 10.2 rv64i also carried with it _zicsr. This is no longer the case with gcc 12 and higher.

If a DUT outside of the test code (model.h) reads/writes a csr it will fail at compile under gcc 12. The -march is passed into the python prog/script that generates the Makefile. For example under I extension it uses rv64i.

If the DUT changes this in the python code to be hardcoded to some other value, this may and can modify the DUT generated code vs the Sail generated code resulting in a signature diff between the two.

I ran into this as I hardcoded the python to be -march=rv64imafdc_zicsr_zifencei. The jalr and jal opcodes under I failed. I could not figure out why as both codes if you followed the code execution came up with correct signatures for themselves but different for DUT and Sail.

I then noticed that the DUT code had compressed opcodes in it the Sail code did not. This was causing address differences that would make the signatures calulated based on the addresses in the code different.

— Reply to this email directly, view it on GitHub < https://url.avanan.click/v2/___https://github.com/riscv-non-isa/riscv-arch-test/issues/369 < https://url.avanan.click/v2/___https:/github.com/riscv-non-isa/riscv-arch-test/issues/369>>___.YXAzOmluc3BpcmVzZW1pOmE6bzoyMzA4ZjNlODM1Nzg3NjY1YjkwYWVlNzlkMjc2N2JkMDo2OjFmNjk6NGExODUyYWE0MjZlYzBiMjA0NzhjNTY5MjI4M2ZiMmQ4MDAzZDM5Y2RhYjFiMTRiNDE1YjAzNDVmODU2ODM0MDp0OlQ;, or unsubscribe < https://url.avanan.click/v2/___https://github.com/notifications/unsubscribe-auth/AHPXVJVROWV5XSVC3RULADLXR5AXNANCNFSM6AAAAAA2WOS53E < https://url.avanan.click/v2/___https:/github.com/notifications/unsubscribe-auth/AHPXVJVROWV5XSVC3RULADLXR5AXNANCNFSM6AAAAAA2WOS53E>>___.YXAzOmluc3BpcmVzZW1pOmE6bzoyMzA4ZjNlODM1Nzg3NjY1YjkwYWVlNzlkMjc2N2JkMDo2OmFjZTk6OTVhZjY5ZGRhNTdiZjUyNTk5MDkzMjdkZWZjYjE1MGI3NTQ0NTQ2YjA0YTY4MjFlOTQ4MGNlMTAyNTIwM2U1YTp0OlQ;

. You are receiving this because you are subscribed to this thread.Message ID: @.<mailto:@.>>

— Reply to this email directly, view it on GitHub< https://url.avanan.click/v2/___https:/github.com/riscv-non-isa/riscv-arch-test/issues/369%23issuecomment-1650258617___.YXAzOmluc3BpcmVzZW1pOmE6bzoyMzA4ZjNlODM1Nzg3NjY1YjkwYWVlNzlkMjc2N2JkMDo2OmIzYzk6NWM5ZWU0ZjkzMTg1NTFhOWM5N2UwZmU1ZTQ2YmI4NzY3MjYyZWIwNDk1NTI2OTQzZWQzOTdkOGJlMGUwZTdmYTpoOlQ>, or unsubscribe< https://url.avanan.click/v2/___https:/github.com/notifications/unsubscribe-auth/AR3S6Y5RWLGIZREBL5VLUNTXR774LANCNFSM6AAAAAA2WOS53E___.YXAzOmluc3BpcmVzZW1pOmE6bzoyMzA4ZjNlODM1Nzg3NjY1YjkwYWVlNzlkMjc2N2JkMDo2OmE3MTk6MWRmZWNjNDJiOWZiYTU5MzY5OWQyZjZkM2IwMzJlODdjMDk1OWMyYzA5YThmZWE5NTdkODI0OTlmMGJjZjY0NjpoOlQ>.

You are receiving this because you authored the thread.Message ID: @.**@.>>

— Reply to this email directly, view it on GitHub https://github.com/riscv-non-isa/riscv-arch-test/issues/369#issuecomment-1650603636, or unsubscribe https://github.com/notifications/unsubscribe-auth/AHPXVJTTING2XDTMQK37JFTXSA46NANCNFSM6AAAAAA2WOS53E . You are receiving this because you commented.Message ID: @.***>

[InspireSemi]

It was not in the trap handler.. It was in one of the test cases.

Under gcc 10.2 whne you specify rv64i for -march you will automatically also get _zcsr (at a minimum, I think there are some other assumed extensions like _zfencei) In our DUT code in the pre test and post test code that is in our model.h we are reading some csr registers as part of our DUT init. When compiling under gcc10.2 to test I extension this was all good, both Sail and DUT got rv64i, our csr opcodes were ok.

I was looking to move our compiler to gcc12.2, first test was to run riscof with this cross-compiler. During the compile of the DUT tests, it errored out, saying that zcsr needed to be defined. In debugging this, I found out that gcc 12.2 NO LONGER ASSUMES _zcsr when i is in the -march, so -march=rv64i is just I and nothing else.

To fix this I just put the full string for -march=rv64imacdf_zcsr_zfencei in our python code that generates the DUT makefile. Was able to compile and run but jal and jalr under I are failing.

In debugging this, I found that because c was part of the string for the -march for the DUT, most of the test cases were being optimized with compressed opcodes. (Mostly nops) The Sail model in compiling its code was just using -march=rv64i and had no compressed opcodes in it. This was causing the addresses to be slightly different between DUT and Sail resulting in a different signature for the jal / jal tests.

Went back to gcc10.2 and removed my fix for -march= in python code so that it was once again using what was passed to it, rv64i. Failure conditions went away, DUT and Sail signatures matched now.

The issue is since riscof lets you define code before and after the tests in model.h for custom init() and cleanup() type code. The DUT may need like we did _zcsr or some other extension, that could like adding the c extension cause the generated code between DUT and Sail to be different (like it did with us) and the signatures different.

Hope this explains what I am concerned about.

I think I can fix this by adding a _zcsr to the python script where it uses the value passed into it. The -march would be -march=rv64i_zcsr. Not sure this will not cause other problems under gcc 12. Marc Karasek Principal Software Engineer M: 678.770.3788

[A close up of a sign Description automatically generated] www.inspiresemi.comhttp://www.cryptocoretech.com/

THIS MESSAGE IS INTENDED ONLY FOR THE USE OF THE INDIVIDUAL OR ENTITY TO WHICH IT IS ADDRESSED AND MAY CONTAIN INFORMATION THAT IS PRIVILEGED, CONFIDENTIAL AND/OR EXEMPT FROM DISCLOSURE UNDER APPLICABLE LAW. If the reader of this message is not the intended recipient or agent responsible for delivering the message to the intended recipient, then you are hereby notified that any dissemination or copying of this communication is strictly prohibited. If you have received this electronic transmission in error, please delete it from your system without copying it and notify the sender by reply e-mail so that our address record can be corrected. Thank you.

From: Allen Baum @.> Sent: Tuesday, July 25, 2023 8:15 PM To: riscv-non-isa/riscv-arch-test @.> Cc: Marc Karasek @.>; Author @.> Subject: Re: [riscv-non-isa/riscv-arch-test] With newer compiler -march options may cause issues between DUT and Sail (Issue #369)

GCC shouldn't be able to use compressed ops in the trap handler because it specifically has .option norvc everywhere (except for that one exception). I am confused if this is a bug in GCC, in our tests, in riscof, in Sail, or in your test configuration? If you pass the C extension in -march string, the tools should allow compressed ops to be generated - including for Sail. Why was Sail getting a different compile parameters than your DUT? Were you using GCC12 for one of them and GCC10 for the other?

On Tue, Jul 25, 2023 at 2:45 PM InspireSemi @.<mailto:@.>> wrote:

It is not how careful you are the, gcc optimized the code to use compressed because I told It that it had Compressed available (set the march to rv64imafdc_zcsr_zfencei) while the sail model got just the rv64i for its compile.

I saw in the two objdumps one from out DUT and one from Sail that our code had compressed opcodes in it sail did not. This was in the test case code. I reverted my change back so the python code was again using the passed in march parameter. Made sure I was compiling with gcc10.2 and magic signatures matched.

What this caused was the DUT to have one signature and Sail to have another signature. This was because the signatures are dependent on the code having the same opcodes. As soon as one side has a 16 bit opcode (for example nop) it throws the signature calculation off.

The issue that started this was I was looking to move internally to gcc12. Tried to run riscof under this. It was failing to compile because it said I had a csr opcode and no support for it??? Found out the under gcc10.2 rv64i implied _zcsr for march. This was changed under gcc 12.x and rv64i is just I now. I made the change above to get it to run under gcc 12 and hardcoded the march in the python code to cover all cases.

This then led to this bug in mismatched signatures.

Marc

Marc Karasek Principal Software Engineer M: 678.770.3788

[A close up of a sign Description automatically generated] https://url.avanan.click/v2/___www.inspiresemi.comhttp://www.cryptocoretech.com/.YXAzOmluc3BpcmVzZW1pOmE6bzpjMWIzNzJhNGVmM2M1MWIwOWRmYjEzMTEyOTJlNTNjYjo2Ojc3ZDI6MDcyZGYwMTJkNWNjMmVhMWQ3OGRlZmZjYTUzZTM0NTM1NTdkNDZhNGMzOGQ4NjhmZmM2MGRhN2RhODk2NWY0MTp0OlQ<https://url.avanan.click/v2/www.inspiresemi.com%3chttp:/www.cryptocoretech.com/%3e___.YXAzOmluc3BpcmVzZW1pOmE6bzpjMWIzNzJhNGVmM2M1MWIwOWRmYjEzMTEyOTJlNTNjYjo2Ojc3ZDI6MDcyZGYwMTJkNWNjMmVhMWQ3OGRlZmZjYTUzZTM0NTM1NTdkNDZhNGMzOGQ4NjhmZmM2MGRhN2RhODk2NWY0MTp0OlQ>;

THIS MESSAGE IS INTENDED ONLY FOR THE USE OF THE INDIVIDUAL OR ENTITY TO WHICH IT IS ADDRESSED AND MAY CONTAIN INFORMATION THAT IS PRIVILEGED, CONFIDENTIAL AND/OR EXEMPT FROM DISCLOSURE UNDER APPLICABLE LAW. If the reader of this message is not the intended recipient or agent responsible for delivering the message to the intended recipient, then you are hereby notified that any dissemination or copying of this communication is strictly prohibited. If you have received this electronic transmission in error, please delete it from your system without copying it and notify the sender by reply e-mail so that our address record can be corrected. Thank you.

From: Allen Baum @.<mailto:@.>> Sent: Tuesday, July 25, 2023 1:37 PM To: riscv-non-isa/riscv-arch-test @.<mailto:@.>> Cc: Marc Karasek @.<mailto:@.>>; Author @.<mailto:@.>> Subject: Re: [riscv-non-isa/riscv-arch-test] With newer compiler -march options may cause issues between DUT and Sail (Issue #369)

Were the compressed ops in the tests themselves or in the trap handler? We were pretty careful in the trap handler to ensure that it used no compressed ops (except as c.nop as padding, IIRC)

On Mon, Jul 24, 2023 at 9:05 PM InspireSemi @.<mailto:@.mailto:***@***.***%3cmailto:***@***.***>>

wrote:

Sometime after gcc 10.2, the -march option has changed. For 10.2 rv64i also carried with it _zicsr. This is no longer the case with gcc 12 and higher.

If a DUT outside of the test code (model.h) reads/writes a csr it will fail at compile under gcc 12. The -march is passed into the python prog/script that generates the Makefile. For example under I extension it uses rv64i.

If the DUT changes this in the python code to be hardcoded to some other value, this may and can modify the DUT generated code vs the Sail generated code resulting in a signature diff between the two.

I ran into this as I hardcoded the python to be -march=rv64imafdc_zicsr_zifencei. The jalr and jal opcodes under I failed. I could not figure out why as both codes if you followed the code execution came up with correct signatures for themselves but different for DUT and Sail.

I then noticed that the DUT code had compressed opcodes in it the Sail code did not. This was causing address differences that would make the signatures calulated based on the addresses in the code different.

— Reply to this email directly, view it on GitHub < https://url.avanan.click/v2/___https://github.com/riscv-non-isa/riscv-arch-test/issues/369https://url.avanan.click/v2/___https:/github.com/riscv-non-isa/riscv-arch-test/issues/369 < https://url.avanan.click/v2/___https:/github.com/riscv-non-isa/riscv-arch-test/issues/369>>___.YXAzOmluc3BpcmVzZW1pOmE6bzoyMzA4ZjNlODM1Nzg3NjY1YjkwYWVlNzlkMjc2N2JkMDo2OjFmNjk6NGExODUyYWE0MjZlYzBiMjA0NzhjNTY5MjI4M2ZiMmQ4MDAzZDM5Y2RhYjFiMTRiNDE1YjAzNDVmODU2ODM0MDp0OlQhttps://url.avanan.click/v2/___https:/github.com/riscv-non-isa/riscv-arch-test/issues/369%3e%3e___.YXAzOmluc3BpcmVzZW1pOmE6bzoyMzA4ZjNlODM1Nzg3NjY1YjkwYWVlNzlkMjc2N2JkMDo2OjFmNjk6NGExODUyYWE0MjZlYzBiMjA0NzhjNTY5MjI4M2ZiMmQ4MDAzZDM5Y2RhYjFiMTRiNDE1YjAzNDVmODU2ODM0MDp0OlQ;, or unsubscribe < https://url.avanan.click/v2/___https://github.com/notifications/unsubscribe-auth/AHPXVJVROWV5XSVC3RULADLXR5AXNANCNFSM6AAAAAA2WOS53Ehttps://url.avanan.click/v2/___https:/github.com/notifications/unsubscribe-auth/AHPXVJVROWV5XSVC3RULADLXR5AXNANCNFSM6AAAAAA2WOS53E < https://url.avanan.click/v2/___https:/github.com/notifications/unsubscribe-auth/AHPXVJVROWV5XSVC3RULADLXR5AXNANCNFSM6AAAAAA2WOS53E>>___.YXAzOmluc3BpcmVzZW1pOmE6bzoyMzA4ZjNlODM1Nzg3NjY1YjkwYWVlNzlkMjc2N2JkMDo2OmFjZTk6OTVhZjY5ZGRhNTdiZjUyNTk5MDkzMjdkZWZjYjE1MGI3NTQ0NTQ2YjA0YTY4MjFlOTQ4MGNlMTAyNTIwM2U1YTp0OlQhttps://url.avanan.click/v2/___https:/github.com/notifications/unsubscribe-auth/AHPXVJVROWV5XSVC3RULADLXR5AXNANCNFSM6AAAAAA2WOS53E%3e%3e___.YXAzOmluc3BpcmVzZW1pOmE6bzoyMzA4ZjNlODM1Nzg3NjY1YjkwYWVlNzlkMjc2N2JkMDo2OmFjZTk6OTVhZjY5ZGRhNTdiZjUyNTk5MDkzMjdkZWZjYjE1MGI3NTQ0NTQ2YjA0YTY4MjFlOTQ4MGNlMTAyNTIwM2U1YTp0OlQ;

. You are receiving this because you are subscribed to this thread.Message ID: @.<mailto:@.mailto:***@***.***%3cmailto:***@***.***>>

— Reply to this email directly, view it on GitHub< https://url.avanan.click/v2/___https:/github.com/riscv-non-isa/riscv-arch-test/issues/369%23issuecomment-1650258617___.YXAzOmluc3BpcmVzZW1pOmE6bzoyMzA4ZjNlODM1Nzg3NjY1YjkwYWVlNzlkMjc2N2JkMDo2OmIzYzk6NWM5ZWU0ZjkzMTg1NTFhOWM5N2UwZmU1ZTQ2YmI4NzY3MjYyZWIwNDk1NTI2OTQzZWQzOTdkOGJlMGUwZTdmYTpoOlQ>, or unsubscribe< https://url.avanan.click/v2/___https:/github.com/notifications/unsubscribe-auth/AR3S6Y5RWLGIZREBL5VLUNTXR774LANCNFSM6AAAAAA2WOS53E___.YXAzOmluc3BpcmVzZW1pOmE6bzoyMzA4ZjNlODM1Nzg3NjY1YjkwYWVlNzlkMjc2N2JkMDo2OmE3MTk6MWRmZWNjNDJiOWZiYTU5MzY5OWQyZjZkM2IwMzJlODdjMDk1OWMyYzA5YThmZWE5NTdkODI0OTlmMGJjZjY0NjpoOlQ>.

You are receiving this because you authored the thread.Message ID: @.**@.mailto:***@***.******@***.***>>

— Reply to this email directly, view it on GitHub https://url.avanan.click/v2/___https://github.com/riscv-non-isa/riscv-arch-test/issues/369%23issuecomment-1650603636<https://url.avanan.click/v2/___https:/github.com/riscv-non-isa/riscv-arch-test/issues/369%23issuecomment-1650603636>.YXAzOmluc3BpcmVzZW1pOmE6bzpjMWIzNzJhNGVmM2M1MWIwOWRmYjEzMTEyOTJlNTNjYjo2OjhiNDk6ZDIxODExZDQ0NWU5NmQwMzFiM2ZiMTM1MGE0ZWVmY2NlY2I0NTRlOTg4ODI3MDg2YzljZDgyMzA4MTNjNzExNDp0OlQ;, or unsubscribe <https://url.avanan.click/v2/https://github.com/notifications/unsubscribe-auth/AHPXVJTTING2XDTMQK37JFTXSA46NANCNFSM6AAAAAA2WOS53Ehttps://url.avanan.click/v2/___https:/github.com/notifications/unsubscribe-auth/AHPXVJTTING2XDTMQK37JFTXSA46NANCNFSM6AAAAAA2WOS53E>___.YXAzOmluc3BpcmVzZW1pOmE6bzpjMWIzNzJhNGVmM2M1MWIwOWRmYjEzMTEyOTJlNTNjYjo2OmQxM2M6MmUxNjYxN2YyNjI5ZGY2MjQ5OTU4OTBkMzMxNzhhOGNhODUxMjc0ZGE1ZTdmODI5ODI2Mjc0N2VjOGE5ZWI3Mjp0OlQ; . You are receiving this because you commented.Message ID: @.<mailto:@.>>

— Reply to this email directly, view it on GitHubhttps://url.avanan.click/v2/___https:/github.com/riscv-non-isa/riscv-arch-test/issues/369%23issuecomment-1650734634___.YXAzOmluc3BpcmVzZW1pOmE6bzpjMWIzNzJhNGVmM2M1MWIwOWRmYjEzMTEyOTJlNTNjYjo2OjBmN2M6MTAxN2JmMTAyYTQwMDE5ODRlYTE5NTVhMDljZWRmY2NjMjMxYTRmMWMxZTYzZjZlNDhlZjdhNzY5NmQxZWE1OTpoOlQ, or unsubscribehttps://url.avanan.click/v2/___https:/github.com/notifications/unsubscribe-auth/AR3S6YZMPISMVPVNSWGUXLLXSBOR5ANCNFSM6AAAAAA2WOS53E___.YXAzOmluc3BpcmVzZW1pOmE6bzpjMWIzNzJhNGVmM2M1MWIwOWRmYjEzMTEyOTJlNTNjYjo2OjYwMjA6ZWY3YmMzZTBkNmI5NjczMWRkMWU0MTA5NjY3MTM0YWU3Nzg4YWMzYTllZjYxOGUzNWFkYmM4MjhiZjBlZDg4NjpoOlQ. You are receiving this because you authored the thread.Message ID: @.**@.>>

[allenjbaum]

I'm still missing something. If the scripts have two different -march strings, then of course I would expect signatures to mismatch. If the models are built with 2 different toolchains, I'd also be unsurprised if there were mismatches We make no guarantees that using different toolchains will work at all. So is this a gcc10 vs gcc12 problem - 2 different tool chains, or just one? Have you been able to make this work with GCC12 for both DUT and Sail or not?

Is this an issue that the (model specific) init() and cleanup() functions don't maintain the .option norvc state that was present when they were entered? If they don't - that indicates a bug in those functions (and possibly in the documentation that should explain that it is a requirement)

I think that you needed to add _zcsr and possibly _zfencei to the -march string of your DUT of GCC10 vs. GCC12 differences. But, whatever you added - why isn't that getting passed to Sail? Or maybe: why isn't the C present in the sail -march string if it is your DUT string?

On Tue, Jul 25, 2023 at 6:18 PM InspireSemi @.***> wrote:

It was not in the trap handler.. It was in one of the test cases.

Under gcc 10.2 whne you specify rv64i for -march you will automatically also get _zcsr (at a minimum, I think there are some other assumed extensions like _zfencei) In our DUT code in the pre test and post test code that is in our model.h we are reading some csr registers as part of our DUT init. When compiling under gcc10.2 to test I extension this was all good, both Sail and DUT got rv64i, our csr opcodes were ok.

I was looking to move our compiler to gcc12.2, first test was to run riscof with this cross-compiler. During the compile of the DUT tests, it errored out, saying that zcsr needed to be defined. In debugging this, I found out that gcc 12.2 NO LONGER ASSUMES _zcsr when i is in the -march, so -march=rv64i is just I and nothing else.

To fix this I just put the full string for -march=rv64imacdf_zcsr_zfencei in our python code that generates the DUT makefile. Was able to compile and run but jal and jalr under I are failing.

In debugging this, I found that because c was part of the string for the -march for the DUT, most of the test cases were being optimized with compressed opcodes. (Mostly nops) The Sail model in compiling its code was just using -march=rv64i and had no compressed opcodes in it. This was causing the addresses to be slightly different between DUT and Sail resulting in a different signature for the jal / jal tests.

Went back to gcc10.2 and removed my fix for -march= in python code so that it was once again using what was passed to it, rv64i. Failure conditions went away, DUT and Sail signatures matched now.

The issue is since riscof lets you define code before and after the tests in model.h for custom init() and cleanup() type code. The DUT may need like we did _zcsr or some other extension, that could like adding the c extension cause the generated code between DUT and Sail to be different (like it did with us) and the signatures different.

Hope this explains what I am concerned about.

I think I can fix this by adding a _zcsr to the python script where it uses the value passed into it. The -march would be -march=rv64i_zcsr. Not sure this will not cause other problems under gcc 12. Marc Karasek Principal Software Engineer M: 678.770.3788

Message ID: @.*** com>

[InspireSemi]

It is not two toolchains..

Running the riscof using gcc12 as the cross compiler. This does not work for our DUT with the passed in -march variable to the python script used to generate the makefile.\ This is because for I the passed in argument is rv64i. Using gcc12 this is just the I extension and nothing else.

Previously under gcc10 rv64i expanded to rv64i_zcsr.

Our DUT code as part of our initialization code has some csr opcodes. Using gcc12 the compile of the test cases for our DUT failed because under gcc12 rv64i does NOT imply also _zcsr.

What I had done to fix this was just hardcode a full -march string in the python script (-march=rv65imcadf_zcsr_zfencei). This allowed the code to compile but failed signature checking.

My concern is with gcc now changing the way -march is defined, and I understand there are more changes coming.. Having a fixed march string of just rv64i for example may not work for a DUT that has code outside of the test cases (init/finish) that needs another extension beyond I. This change to the way -march string is handled in gcc just brought this problem to light.

I am going to have to make changes such that our DUT and Sail will have different -march strings. For I extension I think it will be rv64i for Sail and rv641_zcsr for our DUT. I am hoping this will work. I think all we need is the zcsr extension defined.

I will try my best to stick my head in the door on Thursday meeting to help clear up any confusion. Marc Karasek Principal Software Engineer M: 678.770.3788

[A close up of a sign Description automatically generated] www.inspiresemi.comhttp://www.cryptocoretech.com/

THIS MESSAGE IS INTENDED ONLY FOR THE USE OF THE INDIVIDUAL OR ENTITY TO WHICH IT IS ADDRESSED AND MAY CONTAIN INFORMATION THAT IS PRIVILEGED, CONFIDENTIAL AND/OR EXEMPT FROM DISCLOSURE UNDER APPLICABLE LAW. If the reader of this message is not the intended recipient or agent responsible for delivering the message to the intended recipient, then you are hereby notified that any dissemination or copying of this communication is strictly prohibited. If you have received this electronic transmission in error, please delete it from your system without copying it and notify the sender by reply e-mail so that our address record can be corrected. Thank you.

From: Allen Baum @.> Sent: Wednesday, July 26, 2023 2:23 AM To: riscv-non-isa/riscv-arch-test @.> Cc: Marc Karasek @.>; Author @.> Subject: Re: [riscv-non-isa/riscv-arch-test] With newer compiler -march options may cause issues between DUT and Sail (Issue #369)

I'm still missing something. If the scripts have two different -march strings, then of course I would expect signatures to mismatch. If the models are built with 2 different toolchains, I'd also be unsurprised if there were mismatches We make no guarantees that using different toolchains will work at all. So is this a gcc10 vs gcc12 problem - 2 different tool chains, or just one? Have you been able to make this work with GCC12 for both DUT and Sail or not?

Is this an issue that the (model specific) init() and cleanup() functions don't maintain the .option norvc state that was present when they were entered? If they don't - that indicates a bug in those functions (and possibly in the documentation that should explain that it is a requirement)

I think that you needed to add _zcsr and possibly _zfencei to the -march string of your DUT of GCC10 vs. GCC12 differences. But, whatever you added - why isn't that getting passed to Sail? Or maybe: why isn't the C present in the sail -march string if it is your DUT string?

On Tue, Jul 25, 2023 at 6:18 PM InspireSemi @.<mailto:@.>> wrote:

It was not in the trap handler.. It was in one of the test cases.

Under gcc 10.2 whne you specify rv64i for -march you will automatically also get _zcsr (at a minimum, I think there are some other assumed extensions like _zfencei) In our DUT code in the pre test and post test code that is in our model.h we are reading some csr registers as part of our DUT init. When compiling under gcc10.2 to test I extension this was all good, both Sail and DUT got rv64i, our csr opcodes were ok.

I was looking to move our compiler to gcc12.2, first test was to run riscof with this cross-compiler. During the compile of the DUT tests, it errored out, saying that zcsr needed to be defined. In debugging this, I found out that gcc 12.2 NO LONGER ASSUMES _zcsr when i is in the -march, so -march=rv64i is just I and nothing else.

To fix this I just put the full string for -march=rv64imacdf_zcsr_zfencei in our python code that generates the DUT makefile. Was able to compile and run but jal and jalr under I are failing.

In debugging this, I found that because c was part of the string for the -march for the DUT, most of the test cases were being optimized with compressed opcodes. (Mostly nops) The Sail model in compiling its code was just using -march=rv64i and had no compressed opcodes in it. This was causing the addresses to be slightly different between DUT and Sail resulting in a different signature for the jal / jal tests.

Went back to gcc10.2 and removed my fix for -march= in python code so that it was once again using what was passed to it, rv64i. Failure conditions went away, DUT and Sail signatures matched now.

The issue is since riscof lets you define code before and after the tests in model.h for custom init() and cleanup() type code. The DUT may need like we did _zcsr or some other extension, that could like adding the c extension cause the generated code between DUT and Sail to be different (like it did with us) and the signatures different.

Hope this explains what I am concerned about.

I think I can fix this by adding a _zcsr to the python script where it uses the value passed into it. The -march would be -march=rv64i_zcsr. Not sure this will not cause other problems under gcc 12. Marc Karasek Principal Software Engineer M: 678.770.3788

Message ID: @.<mailto:@.> com>

— Reply to this email directly, view it on GitHubhttps://url.avanan.click/v2/___https:/github.com/riscv-non-isa/riscv-arch-test/issues/369%23issuecomment-1651051896___.YXAzOmluc3BpcmVzZW1pOmE6bzoxMGQ3M2Q4MjRjMjA3ZDA3MDJmMTVjZDEzNzU3MGY4OTo2OjQzMWM6NTc0NTUxMjU4MWYzMmNiZGY0ZDYyOTMyYWYyMGIyNjYxZTJmOGY3NWYzYWNjMDJhMGJkNDBmYmE2ZTNiZmFjMzpoOlQ, or unsubscribehttps://url.avanan.click/v2/___https:/github.com/notifications/unsubscribe-auth/AR3S6YYERKVE4EATOEGHA63XSCZTVANCNFSM6AAAAAA2WOS53E___.YXAzOmluc3BpcmVzZW1pOmE6bzoxMGQ3M2Q4MjRjMjA3ZDA3MDJmMTVjZDEzNzU3MGY4OTo2OjkxM2Q6ZmFhNGY0ZGI1YzcxNWMyNDA3NzBmNDM1OTE2MTgxYTE4NjUzNzIzNWRkMTJkMjc3ZDY5ZWI2MjM2OWIzODZmZTpoOlQ. You are receiving this because you authored the thread.Message ID: @.**@.>>

[allenjbaum]

This still isn't making sense. If your DUT supports Zicsr, why isn't that explicitly passed to both? Why can't you have rv64i_Zicsr for both sail and DUT? I understand the toolchains have changed handling Zicsr - but if it's explicitly passed, that shouldn't matter. If passing -march=rv65imcadf_zcsr_zfencei failed because you model specific init() and cleanup() had C ops in them, why doesn't {.push; .option rvc, ...... .align 2; .pop} surrounding your functions fix that? In general, it shouldn't matter if DUT uses code outside the test case in those custom functions if Sail won't be using it. In general, I thought tests started with .option norvc to avoid this problem. But regardless, as long as tests are compiled with that C option being set independently of model specific macros. If the model specific macros change that option, then they need to change it back (as in the example above)

On Wed, Jul 26, 2023 at 6:08 AM InspireSemi @.***> wrote:

It is not two toolchains..

Running the riscof using gcc12 as the cross compiler. This does not work for our DUT with the passed in -march variable to the python script used to generate the makefile.\ This is because for I the passed in argument is rv64i. Using gcc12 this is just the I extension and nothing else.

Previously under gcc10 rv64i expanded to rv64i_zcsr.

Our DUT code as part of our initialization code has some csr opcodes. Using gcc12 the compile of the test cases for our DUT failed because under gcc12 rv64i does NOT imply also _zcsr.

What I had done to fix this was just hardcode a full -march string in the python script (-march=rv65imcadf_zcsr_zfencei). This allowed the code to compile but failed signature checking.

My concern is with gcc now changing the way -march is defined, and I understand there are more changes coming.. Having a fixed march string of just rv64i for example may not work for a DUT that has code outside of the test cases (init/finish) that needs another extension beyond I. This change to the way -march string is handled in gcc just brought this problem to light.

I am going to have to make changes such that our DUT and Sail will have different -march strings. For I extension I think it will be rv64i for Sail and rv641_zcsr for our DUT. I am hoping this will work. I think all we need is the zcsr extension defined.

Message ID: @.*** com>

[InspireSemi]

It is passed from riscof to the python script. I am not doing it riscof is passing this parameter..

Sent from my T-Mobile 5G Device Get Outlook for Androidhttps://aka.ms/AAb9ysg

---

From: Allen Baum @.> Sent: Wednesday, July 26, 2023 12:56:16 PM To: riscv-non-isa/riscv-arch-test @.> Cc: Marc Karasek @.>; Author @.> Subject: Re: [riscv-non-isa/riscv-arch-test] With newer compiler -march options may cause issues between DUT and Sail (Issue #369)

This still isn't making sense. If your DUT supports Zicsr, why isn't that explicitly passed to both? Why can't you have rv64i_Zicsr for both sail and DUT? I understand the toolchains have changed handling Zicsr - but if it's explicitly passed, that shouldn't matter. If passing -march=rv65imcadf_zcsr_zfencei failed because you model specific init() and cleanup() had C ops in them, why doesn't {.push; .option rvc, ...... .align 2; .pop} surrounding your functions fix that? In general, it shouldn't matter if DUT uses code outside the test case in those custom functions if Sail won't be using it. In general, I thought tests started with .option norvc to avoid this problem. But regardless, as long as tests are compiled with that C option being set independently of model specific macros. If the model specific macros change that option, then they need to change it back (as in the example above)

On Wed, Jul 26, 2023 at 6:08 AM InspireSemi @.***> wrote:

It is not two toolchains..

Running the riscof using gcc12 as the cross compiler. This does not work for our DUT with the passed in -march variable to the python script used to generate the makefile.\ This is because for I the passed in argument is rv64i. Using gcc12 this is just the I extension and nothing else.

Previously under gcc10 rv64i expanded to rv64i_zcsr.

Our DUT code as part of our initialization code has some csr opcodes. Using gcc12 the compile of the test cases for our DUT failed because under gcc12 rv64i does NOT imply also _zcsr.

What I had done to fix this was just hardcode a full -march string in the python script (-march=rv65imcadf_zcsr_zfencei). This allowed the code to compile but failed signature checking.

My concern is with gcc now changing the way -march is defined, and I understand there are more changes coming.. Having a fixed march string of just rv64i for example may not work for a DUT that has code outside of the test cases (init/finish) that needs another extension beyond I. This change to the way -march string is handled in gcc just brought this problem to light.

I am going to have to make changes such that our DUT and Sail will have different -march strings. For I extension I think it will be rv64i for Sail and rv641_zcsr for our DUT. I am hoping this will work. I think all we need is the zcsr extension defined.

Message ID: @.*** com>

— Reply to this email directly, view it on GitHubhttps://url.avanan.click/v2/___https://github.com/riscv-non-isa/riscv-arch-test/issues/369%23issuecomment-1652182436___.YXAzOmluc3BpcmVzZW1pOmE6bzplNGQ2MmIzNTRkMzljYTJjNmU0ZmYzYTBmNzVhYTU0MDo2OmZlYjA6ODQ3Y2VhYzkzZjAxYmViOTJlY2I1YzRiYTc4MDRhMGE2MjBmZmYxYWE0ZTQwMTIyYTllYzVhMjhjNjgzOGMxMjpoOlQ, or unsubscribehttps://url.avanan.click/v2/___https://github.com/notifications/unsubscribe-auth/AR3S6Y4ND6PYR4MRAMX6E43XSFD3BANCNFSM6AAAAAA2WOS53E___.YXAzOmluc3BpcmVzZW1pOmE6bzplNGQ2MmIzNTRkMzljYTJjNmU0ZmYzYTBmNzVhYTU0MDo2OjQxZjM6OTc4MDI0M2E4MTU4OWQzMjllMTQxMTNlMTE5ZDIwNjlmYTRlNTAzOTNlZDA3MzhkYTg5ZmRiZjk3OTVhNGNlMjpoOlQ. You are receiving this because you authored the thread.Message ID: @.***>

[asb]

My concern is with gcc now changing the way -march is defined, and I understand there are more changes coming..

(Just a lurker here, but commenting from a Clang/LLVM perspective). The changes to move Zicsr out of RV32I/RV64I were an upstream spec change, and of course gave us a quandary from a toolchain perspective of what to do. If you do spot any planned changes likely to cause issues then please do feedback - much easier to know sooner rather than later if there's something we hadn't anticipated.

For what it's worth, on the Clang/LLVM side we so far allow instructions from zicsr and zifencei to be used unconditionally, even if not specified in the -march string on the basis that it seemed the least worst option available to us to avoid too much pain for end users.

[InspireSemi]

IN gcc 12.2 if you do not specify zcsr on the march it will error out on the compile.. what I observed

Marc Karasek Principal Software Engineer M: 678.770.3788

[A close up of a sign Description automatically generated] www.inspiresemi.comhttp://www.cryptocoretech.com/

THIS MESSAGE IS INTENDED ONLY FOR THE USE OF THE INDIVIDUAL OR ENTITY TO WHICH IT IS ADDRESSED AND MAY CONTAIN INFORMATION THAT IS PRIVILEGED, CONFIDENTIAL AND/OR EXEMPT FROM DISCLOSURE UNDER APPLICABLE LAW. If the reader of this message is not the intended recipient or agent responsible for delivering the message to the intended recipient, then you are hereby notified that any dissemination or copying of this communication is strictly prohibited. If you have received this electronic transmission in error, please delete it from your system without copying it and notify the sender by reply e-mail so that our address record can be corrected. Thank you.

From: Alex Bradbury @.> Sent: Thursday, July 27, 2023 10:44 AM To: riscv-non-isa/riscv-arch-test @.> Cc: Marc Karasek @.>; Author @.> Subject: Re: [riscv-non-isa/riscv-arch-test] With newer compiler -march options may cause issues between DUT and Sail (Issue #369)

My concern is with gcc now changing the way -march is defined, and I understand there are more changes coming..

(Just a lurker here, but commenting from a Clang/LLVM perspective). The changes to move Zicsr out of RV32I/RV64I were an upstream spec change, and of course gave us a quandary from a toolchain perspective of what to do. If you do spot any planned changes likely to cause issues then please do feedback - much easier to know sooner rather than later if there's something we hadn't anticipated.

For what it's worth, on the Clang/LLVM side we so far allow instructions from zicsr and zifencei to be used unconditionally, even if not specified in the -march string on the basis that it seemed the least worst option available to us to avoid too much pain for end users.

— Reply to this email directly, view it on GitHubhttps://url.avanan.click/v2/___https:/github.com/riscv-non-isa/riscv-arch-test/issues/369%23issuecomment-1653765247___.YXAzOmluc3BpcmVzZW1pOmE6bzpjYmU1MTc1NDkxMzdmZTFlYzIxNjVmNWM3MjM5ZTk5ZTo2OjY4MDg6OTE0NzVjZTJiY2JhZTVjM2ZlNmEyOTE2ZjQ1YjkxMTUxMzZlNzUzNGY3OGFiYTA0MzdmZGFkN2ZjMTE4MDdiNjpoOlQ, or unsubscribehttps://url.avanan.click/v2/___https:/github.com/notifications/unsubscribe-auth/AR3S6Y5COQVOAYI7RHQ3ZTTXSJ5BLANCNFSM6AAAAAA2WOS53E___.YXAzOmluc3BpcmVzZW1pOmE6bzpjYmU1MTc1NDkxMzdmZTFlYzIxNjVmNWM3MjM5ZTk5ZTo2OmIxYjU6NWM4M2M4ZDgxY2JlMTAzOTI4OWFiOTI2NjY2NWUwYzQ5ZDUyNmVlNDc1ZDEyN2EzY2IxYzM2OWE1NWNkZjMyOTpoOlQ. You are receiving this because you authored the thread.Message ID: @.**@.>>