Simple Amazon AWS EC2 F1 example

[KoviRobi]

Hi,

I'm fairly new to the FPGA development stuff, I'm just tying to play around with AWS F1, so I have done

$ cd examples/simple
$ make build.awsf1

the output of the make is in https://gist.github.com/KoviRobi/8316a68ee10dc7e3b22b0298ab6d9339 but let me highlight a couple of lines

1. grep: /home/rmk35/programming/bluespec/connectal/boardinfo/.json: No such file or directory, perhaps a bug in Makefile.connectal, or I'm doing something wrong? There is an boardinfo/awsf1.json
2. Do I need quartus/vivado/vsim? I was just hoping to do the gen on my machine, and the synthesise on the Amazon FPGA Developer AMI.
3. make[1]: *** No rule to make target 'obj/Vector.bo', needed by 'obj/AwsF1Top.bo'. Stop. I guess it's looking for the Vector.bo shipped with bluespec, which with the bluespec installed with the Nix package manager is /nix/store/2vxm66aq8f580x90z6nwc4iglm3ahijh-bluespec-unstable-2020.02.09/lib/Libraries/Vector.bo in this instance, but this path (specifically the hash in /nix/store/<hash>-<name>) will change with different versions of bsc so should definitely not be hardcoded.

[jameyhicks]

1. I don't know what is causing this.

2. You need vivado 2019.2 with a license for the vcu9p FPGA synthesis and implementation.

An easy way to get these tools is to launch an AWS FPGA Developer AMI instance. That AMI is based on centos.

3. Connectal still requires BLUESPECDIR to point to the bsc lib directory but has been updated for the layout of libraries in the opensource bsc.

Connectal could use which bsc to determine BLUESPECDIR if it's not set.

[KoviRobi]

Seems like the AWS FPGA Developer AMI only has GHC version 7.6 (and for some reason, it doesn't like installing the Nix package manager) so getting bluespec might be a bit difficult. I wonder if can I do the bluespec/bluesim parts on my machine, and the synthesis parts on the FPGA Developer AMI?

[jameyhicks]

And it's Centos, which I find to be a pain. I had success copying vivado into a similarly configured Ubuntu 18.04 instance. I imagine you could do the same with other Linux distros.

[KoviRobi]

Don't know if that will break any licenses (either break as in stop the software working, or as in legally I shouldn't do that). But we do have a lab machine that has Vivado set up (though not yet with the right license to do the encryption).

Is that a no on the being able to do the bluespec stuff on a separate machine to the synthesis stuff?

[jameyhicks]

The software works but I did not ask if it was OK. I figure I'm complying with the spirit of the license: I only run it on AWS instances and only to develop for AWS FPGAs.

However, it's actually not hard to split those steps: 1) at the point where it attempts to run vivado for synthesis and implementation, all the necessary files are in awsf1/build. You should be able to copy that and the cd awsf1/build/scripts; ./build.sh

I changed some AWS scripts this week, btw.

If this makes sense, we could add a Makefile target to make this easier, or add a remote synthesis option that uses rsync and ssh (or other appropriate mechanism). It might be handy to have it start the instance, run the build, and then shut it down again, at least as an option.

[KoviRobi]

That's great to hear the steps could be split! In the meanwhile I might be having success with installing Nix on CentOS so that I can get the FPGA Developer AMI to have Bluespec.

But still, splitting the steps will allow me to develop/simulate on my laptop, and only be billed for the synthesis time :) (Though I'm still just starting with the FPGA stuff so not 100% sure on how the workflow goes)

Solution that got me Nix installed on CentOS: https://github.com/NixOS/nix/issues/2632#issuecomment-481701310

[jameyhicks]

For simulation, I recommend build.verilator, though it depends on what you'd like to do with AWS.

BSV compile time is short compared to synthesis and implementation for AWS.

Simulation of the AWS CL and SH requires Xilinx xsim or other commercial verilog simulators, which works but requires more manual steps.

[KoviRobi]

Got the problem with 1. (the $(BOARD) not being defined) down:

https://github.com/cambridgehackers/connectal/blob/3f1bf0518b3bac3fa8acb414f3e253d001c829b2/Makefile.connectal#L206 forces the evaluation of the deferred variable $(PCIEGEN), defined at https://github.com/cambridgehackers/connectal/blob/3f1bf0518b3bac3fa8acb414f3e253d001c829b2/Makefile.connectal#L178 which requires $(BOARD) to be defined, but that is only defined at https://github.com/cambridgehackers/connectal/blob/3f1bf0518b3bac3fa8acb414f3e253d001c829b2/Makefile.connectal#L41-L45 after the gen.% target/rule is evaluated, whereas make processes conditional directives before rules.

[jameyhicks]

I'm puzzled why you are getting this, because I've been running make build.awsf1 over and over again the past couple of weeks. Is the behavior the same with make gen.awsf1 ?

[KoviRobi]

Yes, though it's just one line in the top of the Makefile, it still continues on fine. Now that I have figured out why I am getting that, I wonder why you aren't getting it, are you using the same Makefile.connectal as in the master branch here? Note, I'm using GNU Make 4.2.1

[jameyhicks]

GNU Make 4.1.

[KoviRobi]

But this is just a simple thing, make gen.awsf1 BOARD=awsf1 solves this problem.

A bigger thing I'm having is make[1]: *** No rule to make target 'obj/XilinxCells.bo', needed by 'obj/SyncAxisFifo32x8.bo'. Stop.. I couldn't find XilinxCells.* in either $BLUESPECDIR or connectal repo

[jameyhicks]

XilinxCells.bsv is in bsc-contrib

[jameyhicks]

If we clean up the split build, then Travis could run awsf1 builds.

[KoviRobi]

Right, I have I think got to the point where I have done all that is needed on my local machine :) I.e. when it tries to run connectal/scripts/aws/build.sh https://gist.github.com/bc9b3c13971584c14a2b543c1bf1eaa5

Also, I have 'packaged' bsc-contrib as a BlueSpec library for NixOS (inverted commas are there because it's not an upstreamed first class package yet). I think Nix could function as the BlueSpec package manager, but obviously I quite like Nix (Nix doesn't require NixOS, nor specifically Linux :) ).

https://gist.github.com/KoviRobi/1381abdfce8e87139e4d04cfef3d76be#file-example-bluespec-shell-nix-L16-L27

[KoviRobi]

Thanks for your help, I can confirm I have managed to synthesise and deploy the examples/echo program, so closing. Here is a summary of the steps I did, but no doubt it can be simplified further and will do that tomorrow:

1. Create an Amazon AWS FPGA Developer AMI instance
2. Clone aws/aws-fpga
3. Clone from cahbridgehackers repos connectal, fpgamake and buildcache
4. Install nix on the FPGA Developer AMI, see https://github.com/NixOS/nix/issues/2632#issuecomment-606867291 for what else I needed to do before sh <(curl https://nixos.org/nix/install) --no-daemon
5. nix-shell shell.nix where shell.nix contains https://gist.github.com/KoviRobi/1381abdfce8e87139e4d04cfef3d76be (this sets up the bluespec tools)
6. cd connectal/examples/echo
7. make build.awsf1, this will fail before synthesis
8. Convert the symlinks in connectal/examples/echo/awsf1/build/src_post_encryption/ to real files (this is kind of due to Nix, but I will patch the encrypt.tcl file and pull request)
9. Edit the encrypt.tcl to not copy the files, by commenting the Developr section
10. source ~/aws-fpga/hdk_setup.sh, export CONNECTALDIR=~/connectal and export CL_DIR=~/connectal/examples/echo/awsf1
11. ~/aws-fpga/hdk/common/shell_stable/build/scripts/aws_build_dcp_from_cl.sh
12. Upload ~/connectal/examples/echo/awsf1/build/checkpoints/to_aws/*.Developer.tar to s3 and create an FPGA image per https://github.com/aws/aws-fpga/tree/master/hdk as usual.