Create Docker image with Michigan toolchain

[immindich]

See #199.

[immindich]

@austinharris @taru-verma where can I find your team's toolchain? I'm trying to install it on a Docker image that will be provided to the researchers.

[taru-verma]

@immindich What would be the best way to provide you guys with our toolchain?

To build and run apps on our Chisel P1 target, we have the following components:

1. RISC-V 64-bit toolchain; we just used the ones provided in the original GFE documentation. We still build 32-bit apps though.
2. Clang + compiler-rt + RISCV Newlib built with our compiler extensions. There is a build directory with all the binaries and the sysroot.
3. FreeRTOS branched off of develop. This is where we build our app just like the examples (say main_tcp). Please note that we are currently not building with main_fett as discussed here - currently awaiting instructions on that.
4. A tool to encrypt the binary.

Please let us know if you need 1 in addition to 2, 3, and 4.

The binaries themselves (app + AWS/firesim) are provided here.

@austinharris do you have anything to add for the hardware side?

[toddmaustin]

@immindich: is it decided that the researchers will be able to build our challenge application and run it in their own debuggers? That would seem to be quite a bit more power than the typical attacker would have? Also, our mock medical database synthesizes data based on a key, which is in the source code, so to exfiltrate the data, they simply need to run our program in their debugger. We didn't anticipate that the researchers would have this much power.

[immindich]

I think the researchers just need to be able to compile their own binaries.

@taru-verma It seems like the main thing I need is your version of clang and your sysroot. If you have precompiled binaries which work on Debian, then I could use those, otherwise I would need access to the source. I'm not sure whether or not we need the tool for encrypting binaries.

@rtadros-Galois Would you be able to provide more information on what exactly our requirements are?

[taru-verma]

@immindich thanks for the info! I can provide the clang binaries, the sysroot, and the binary for the elf encryption tool. They should all work on Debian.

The elf encryption tool is necessary since our processor runs binaries encrypted under our keys. All that needs to be done is run said tool on the ELF before running it on the P1 processor.

@rtadros-Galois How can I provide this toolchain? I have a tarball I can share.

[rtadros125]

@toddmaustin @immindich (CC @kiniry @mattlebeau-galois @rfoot @Abivin12) The researchers won't have access to the binary running on FreeRTOS. However, they should be able to compile their own codes; right? The initial idea is to provide them with a toolchain so that they can build whatever they want in case they found a way to inject some codes, or to read parts of the memory.

Tbh, I am not very familiar with Morpheus apart from reading one of the papers at some point. If you guys want to discuss this more in the specifics of the toolchain, we can schedule a half-an-hour to decide on the details. We're not going to give the researcher any keys, or give them access to a binary that has hardcoded keys. This is out of question.

The part I need to understand more is: this:

The elf encryption tool is necessary since our processor runs binaries encrypted under our keys. All that needs to be done is run said tool on the ELF before running it on the P1 processor.

Is there an encryption tool that any compiled binary needs to pass first before being able to be executed on the P1?

In a more abstract perspective, this is a demonstrator practice. So if we assume that your processor is indeed in the market. We should assume that the researchers have access to everything this processor, when in the market, will provide. If NOT possessing the toolchain, and NOT being able to build binaries that could be run on your processor makes sense, then, this the message we can present to the researchers in the competition documentation for the Michigan work. That they do not have access to the toolchain. Does this make sense?

[rtadros125]

@taru-verma Since we both have AWS access, maybe S3? Please make sure you give 3635-2728-6999 access to the bucket, and make it visible in us-west-2. Thanks.

[taru-verma]

@rtadros-Galois Can you provide the Canonical ID or the email for granting access to the toolchain bucket? I don't have the permissions to create an IAM role to share using the account ID. Thanks!

[rtadros125]

@taru-verma Canonical ID: e05423247a7f592cfc183625cf148fc4515641dc67b9dbb0d79. Thanks.

[taru-verma]

@rtadros-Galois Can you reconfirm the Canonical ID? I get an Invalid ID error when I'm trying to add access for other AWS accounts. The string seems to be shorter (as compared to ours, don't know if they're supposed to match in length).

The part I need to understand more is: this:

The elf encryption tool is necessary since our processor runs binaries encrypted under our keys. All that needs to be done is run said tool on the ELF before running it on the P1 processor.

Is there an encryption tool that any compiled binary needs to pass first before being able to be executed on the P1?

Our processor has support for code and code pointer encryption, and the generated ELF needs to be encrypted with an elf-parser tool provided with the toolchain.

So for example, if main_blinky is built with FreeRTOS - export PROG=main_blinky BSP=aws USE_CLANG=yes USE_MORPHEUS=yes; make clean && make. Then the researchers will run elf-parser -e main_blinky.elf which will encrypt the binary. This binary can then be run on our processor using ./run_sim ....

I have provided a README with detailed instructions on how to build and run apps for our platform in the toolchain share.

[rtadros125]

@taru-verma Sorry. Here it is e05423247a7f592cfc183625cf148fc4515641dc67b9dbb0d7905fe1896b3382.

[rtadros125]

And you don't want to give this elf-parser to the researchers? This contains some secrets?

[taru-verma]

Okay so @immindich @rtadros-Galois I have shared the Morpheus toolchain - it can be accessed in the bucket s3://morpheus-toolchain-share/morpheus-toolchain-06-16-20.tar.gz. It has instructions as well on how to use it. Please let me know if you have any questions, thanks!

@toddmaustin can confirm regarding sharing elf-parser - not sure if it has some secrets which we want to preserve but we also need that to run apps on the processor.

[toddmaustin]

Yes, the boot time key is in there, which we can change without impact to the system. @taru-verma, I'll slack you a link to the line affected.

[immindich]

@toddmaustin @taru-verma Any update on this?

[rtadros125]

@immindich Please see the last comment in #350. They are working in the newest one. Were you asking about an update or smth else?

[immindich]

@rtadros-Galois I was asking about the status of the toolchain. I hadn't seen the latest comments in the issue you linked.

[taru-verma]

@immindich The Michigan toolchain is available at s3://morpheus-toolchain-share/morpheus-toolchain-06-21-20.tar.gz.

It contains the clang and elf-encryption binaries, the sysroot, and FreeRTOS source with changes for the Morpheus platform. I integrated all changes to FreeRTOS-10.0.1 as of 06-17-20. There is a README in there which details how to build and run apps on our platform.

Please let me know if you have any questions, thanks!

[immindich]

@rtadros-Galois @taru-verma I've built a Docker image with the toolchain installed, although I'm still trying to figure out exactly what we are supposed to be providing to the researchers.

The image currently has the whole LLVM build, the sysroot, and tools like elf-parser. The tarball also has the source for the Michigan team's fork of FreeRTOS, but I'm not sure if it makes sense to include that, since we mostly care about letting them compile their own binaries. Without the FreeRTOS source, there isn't any documentation of what flags to give the compiler. What I'm thinking of doing is adding back in the part of the previous README which went over this.

I'm also curious about the last section of the current README. It mentions the AGFI for the processor that has the same key as this version of the toolchain. Is that something the researchers are supposed to have access to? If not, I will remove that section to avoid confusion.

[taru-verma]

Regarding:

The tarball also has the source for the Michigan team's fork of FreeRTOS, but I'm not sure if it makes sense to include that, since we mostly care about letting them compile their own binaries.

We provided the fork of FreeRTOS because of the following changes:

1. Some ret instructions need to be replaced by jalrcp x0, 0(ra) (specifically in FreeRTOS/Demo/RISC-V_Galois_P1/bsp/boot.S and FreeRTOS/Source/portable/GCC/RISC-V/portASM.S) if using Morpheus.
2. boot.S includes code to read from the RNG device and set up encryption at boot time.
3. A simple Global Object Table section was added in link.ld (which was needed by clang for some reason; researchers may or may not get this error):

   .got : { KEEP (*(.got)) } > dmem

4. Some syscalls were added in FreeRTOS/Demo/RISC-V_Galois_P1/bsp/syscalls.c. (EDIT: These were added for M3DB, and can be excluded. Researchers can add their own syscalls as needed).

You can find specifics on these changes by comparing branch develop with the local branch umich_develop in the FreeRTOS directory. If these changes can be conveyed without the FreeRTOS source, feel free to exclude it. @austinharris can provide more details if you need them.

Regarding:

I'm also curious about the last section of the current README. It mentions the AGFI for the processor that has the same key as this version of the toolchain. Is that something the researchers are supposed to have access to? If not, I will remove that section to avoid confusion.

The code encryption for our processor is done by elf-parser. To decrypt and run the binaries built with this toolchain and encrypted by it, you would need the processor pointed to by the AGFI in the README (since the keys for encryption/decryption will match). Our M3DB sample app is encrypted under a different key and consequently has a different AGFI. @rtadros-Galois can point you to where it is located. We want to limit access to any keys our app might use, hence the different AGFIs.

Regarding:

What I'm thinking of doing is adding back in the part of the previous README which went over this.

That version of the toolchain is not available in the S3 bucket anymore (I removed it to avoid confusion). If you don't have a local copy, please let me know and I'll send that README.

Please let me know if you have any other questions, thanks!

[taru-verma]

Also can you clarify what this means:

letting them compile their own binaries

Wouldn't the researchers run just FreeRTOS binaries on this processor, since it's P1? I don't think we have Linux support (which I believe is only on P2). Thanks!

[rtadros125]

@taru-verma You're right. Compiling their own binaries means they might need to compile a tiny program or a couple of calls to inspect the call/return instructions and such. In case they want to manipulate the queries, and want to know which instructions to send, or how to read what they can get if possible. I cannot be sure how they will do it, but I hope they know what to do so that this exercise becomes beneficial to you guys.

[taru-verma]

Okay, that makes it clearer, thanks! So if these programs are FreeRTOS based, then they would need 1 and 2 and maybe 3 from the changes above. In this case, providing the FreeRTOS fork would just make things easier. In the end I think it's your call!

[jrtc27]

1. Some ret instructions need to be replaced by jalrcp x0, 0(ra) (specifically in FreeRTOS/Demo/RISC-V_Galois_P1/bsp/boot.S and FreeRTOS/Source/portable/GCC/RISC-V/portASM.S) if using Morpheus.

Without knowing about Morpheus I'm confused by it's not all of them. If it is all of them (or jalrcp is the common case), you could consider adding a ret instruction alias predicated on whether you're using Morpheus (and adding the negative predicate to the existing one) in the LLVM Target's TableGen sources and then not need to make so many (or any) such modifications to the FreeRTOS source.

3. A simple Global Object Table section was added in link.ld (which was needed by clang for some reason; researchers may or may not get this error):

Likely you have PIC/PIE enabled, either by explicitly passing one of -fpic/-fPIC/-fpie/-fPIE or by having an implicit default. Given Morpheus does ASLR things, I suspect you have such an implicit default?

[austinharris]

It is all of them except for the return in xPortStartFirstTask since that is returning to an address from the TCB that isn't encrypted, rather than an address encrypted by a jal. We could handle it a different way.

[taru-verma]

Likely you have PIC/PIE enabled, either by explicitly passing one of -fpic/-fPIC/-fpie/-fPIE or by having an implicit default. Given Morpheus does ASLR things, I suspect you have such an implicit default?

We tried these, didn't seem to work. Hence GOT was added. If you can build any of the examples (blinky, tcp, etc.) without GOT and with one of these flags, that should be fine.

[jrtc27]

It is all of them except for the return in xPortStartFirstTask since that is returning to an address from the TCB that isn't encrypted, rather than an address encrypted by a jal. We could handle it a different way.

FWIW, we changed ours to set things up as if it were a trap handler so it could just mret, ie put ra in mepc instead, and set mstatus.MPP/mstatus.MPIE accordingly, and changing it to not set mstatus.MIE (since the sequence is no longer interrupt-safe... that was a fun bug to diagnose). That would solve your problem. But I think this is a good example of why my suggestion to make ret be jalrcp is correct, since this is an edge-case that can be handled by an explicit jr ra (alias for jalr zero, 0(ra)), regardless of whether you adopt our approach or stick with the lone jalr.

[jrtc27]

Likely you have PIC/PIE enabled, either by explicitly passing one of -fpic/-fPIC/-fpie/-fPIE or by having an implicit default. Given Morpheus does ASLR things, I suspect you have such an implicit default?

We tried these, didn't seem to work. Hence GOT was added. If you can build any of the examples (blinky, tcp, etc.) without GOT and with one of these flags, that should be fine.

Those flags would force a GOT, not avoid a GOT, which confirms my hypothesis that you default to PIC/PIE. On RISC-V, it is very difficult to create assembly that accesses the GOT with PIC/PIE disabled (you have to expand the instruction sequence out into explicit %got_pcrel_hi/%pcrel_lo; la and lla will both yield PC-relative addresses instead, and are what RISCVISelLowering uses, and it's only when you enable PIC/PIE that la becomes a load from the GOT).

[mattlebeau-galois]

Intent is to provide these images to Researchers via Synack's portal, not from AWS ECR. @immindich Ref: https://github.com/DARPA-SSITH-Demonstrators/SSITH-FETT-Portal/issues/220

[immindich]

I've updated the image to include everything including the FreeRTOS source. I'm still not entirely clear about the AFI. Will the researchers actually have the credentials necessary to access it?

[rtadros125]

@immindich Not afaik.

[immindich]

Ok, then I'll take out the reference to it in the README, since that might be a bit confusing otherwise.

[andrew-bivin]

AFI access for researchers came up during our sync w/ @taru-verma today. Taru - I missed some nuance in my notes - would researchers need AFI access to compile, or to understand what sort of programs they'd like to compile?

Regardless, I don't have an answer for how we'd share the AFI with the 100+ potential researchers. @mattlebeau-galois could we potentially provision an AWS account generic to Synack that researchers could make use of (just spitballing, that particular idea doesn't sound great)?

[taru-verma]

As per Ramy's comment here from a couple of days ago:

Compiling their own binaries means they might need to compile a tiny program or a couple of calls to inspect the call/return instructions and such.

Researchers can compile their own binaries using just the toolchain. To actually run those binaries, they would need access to the AGFI provided in the toolchain README - since they encryption key in the processor at that AGFI would match the one in our elf-parser tool.

Otherwise if they're just running the M3DB sample application, the M3DB elf file, the associated binaries, and the AGFI ID listed in that should be sufficient.

EDIT: @austinharris and I were just talking - and researchers can build programs with Clang but without Morpheus and run on the default AGFI (the one provided in the run_sim script in the Chisel P1 SW Package). That'll be equivalent to building stuff with the default toolchain but with Clang. I guess that can be used for testing, but it would leave out all the stuff the Morpheus compiler adds. In the end I think it just depends on what the researchers want to build/test/run.

[taru-verma]

@jrtc27 Thanks for the comments! Yes, there was an -fPIC flag being passed in newlib - removing that we can build apps without the GOT in FreeRTOS. We were also able to set up the xPortStartFirstTask function to work as a trap handler with mret and it seems to work fine.

I'm not super clear on:

you could consider adding a ret instruction alias predicated on whether you're using Morpheus (and adding the negative predicate to the existing one) in the LLVM Target's TableGen sources

How can I set this up? We have compiler flags we pass to enable Morpheus stuff. I can just alias it such that all rets are replaced, but curious about how to get it to work with compiler flags. Thanks!

[jrtc27]

I'm not super clear on:

you could consider adding a ret instruction alias predicated on whether you're using Morpheus (and adding the negative predicate to the existing one) in the LLVM Target's TableGen sources

How can I set this up? We have compiler flags we pass to enable Morpheus stuff. I can just alias it such that all rets are replaced, but curious about how to get it to work with compiler flags. Thanks!

RISCVInstrInfo.td has:

def : InstAlias<"ret",                   (JALR      X0,      X1, 0), 4>;

You would change this to:

let Predicates = [NotMorpheus] in
def : InstAlias<"ret",                   (JALR      X0,      X1, 0), 4>;

and where you define your jalrcp add (note the lack of the 4; it's only needed for the JALR aliases to enforce a priority order on them):

let Predicates = [IsMorpheus] in
def : InstAlias<"ret",                   (JALRCP    X0,      X1, 0)>;

filling in whatever names you have for those predicates (which I assume you must have added to RISCV.td alongside the ones for the standard extensions).

[immindich]

One final thing before I send the image to Synack: earlier on in this thread we decided to include the FreeRTOS source code in the image. I just want to double check that this is alright, since this would mean that we are giving the researchers the source code to the FETT P1 applications. It doesn't seem like this is a problem, but I just want to make sure, since that repo is currently private. (cc @rtadros-Galois @Abivin12)

[rtadros125]

@immindich The Michigan FreeRTOS app stack is different from ours; they use their own. The FreeRTOSv10.0.1 kernel and additional components/drivers are open source. So if Michigan want to release the binary, this is something between them and DARPA; whatever they decide, we'd just execute. @austinhroach, do you have any input on this?

[austinhroach]

@rtadros-Galois My understanding is that TA-1 teams are generally free to release source code as they wish. But I'll check with DARPA folks and report back if there are any identified concerns.

[mattlebeau-galois]

AFI access for researchers came up during our sync w/ @taru-verma today. Taru - I missed some nuance in my notes - would researchers need AFI access to compile, or to understand what sort of programs they'd like to compile?

Regardless, I don't have an answer for how we'd share the AFI with the 100+ potential researchers. @mattlebeau-galois could we potentially provision an AWS account generic to Synack that researchers could make use of (just spitballing, that particular idea doesn't sound great)?

@Abivin12 - Hmmm, I don't have enough experience w/the AFI deployment process to answer that, I'm afraid. Another possible thought would be to explore if we can distribute the AFI out of band of the AWS marketplace and provide instructions for how to push it to an F1 'manually', though I'm not actually sure that that's a supported workflow. @kurthopfer Do you have any thoughts/suggestions on this?

[rtadros125]

@mattlebeau-galois AFI IS an Amazon artifact. It has to be on Amazon. Giving the researchers the access to an AFI means one thing: that they can instantiate an F1 instance on their own expenses, flash it with the AFI, and do something with it. Which is not a trivial flow at all. IMHO, it's highly unlikely for the researchers to use the AFIs if we gave them access to them.

[mattlebeau-galois]

Thanks @rtadros-Galois ! Understood. That helps my understanding there.

[andrew-bivin]

@taru-verma @austinharris Unfortunately, sharing the AFI directly with researchers is not feasible logistically or policy wise (Synack researchers identities are obfuscated, and we can't create a "group" AWS account for them or share with them individually).

Could you make your AFI public? That way, no direct sharing would be necessary and researchers could access the AFI through anonymous accounts.

[jameyhicks]

MIT and SRI-Cambridge are using public AFIs.

[austinharris]

@Abivin12 Can you please post full instructions (aws commands) for every user-id, every region, and every attribute I need to modify for the AFI's?

[austinharris]

@Abivin12 Can you please post full instructions (aws commands) for every user-id, every region, and every attribute I need to modify for the AFI's?

Or better yet, can the firesim build scripts be modified so the generated AGFI has every permission you will need?

[rtadros125]

@immindich What is preventing this issue from being closed? Is something left? Sharing the AFI is independent from the docker image afaik, so I am just checking on the docker images. Thanks.

[immindich]

@rtadros-Galois You're right, this should be closed.