Improved XDLRC parsing and much faster device generation

[trharoldsen]

This request contains 3 main components.

1. The reverse wire generation has been moved into the Device Generator. This signifantly speeds up the generation of these connections (2 minutes vs 50 minutes for V6 and should scale better with larger devices). In the process, the ExtendedInfo class has been removed and the reverse wire info is included in the main device structure. The loadExtendedInfo methods in Device still exist but they don't do anything.

As a side effect I testing, I also notice a minor bug in the wire connection generation that was leading to connections being generated for wires that had no sources themselves. This was functionally inconsequential as they came from nowhere and were thus unusable, but they did take up a small bit of additional space and I have fixed the mistake.

2. I've updated the XDLRCParser to present the names of the parsed tokens for context when parsing the XDLRC.

3. Created a compressed serialized XDLRC format. This format compresses the XDLRC files down to sizes in the 50-100 MB range (even for midrange V7 devices) from the 3-20+GB XDLRC file range. it uses techniques similar to the Device generation, but unlike the Device generation, does not throw any information away (the entire XDLRC can be retrieved). This is primarily so we can feasibly maintain a set of xdlrc files for all parts and not have to spend an entire day generating each of them when we need to rebuild a part.

Notes -- I have only run this on Virtex 6 and have not generated the Artix7 parts. If someone could this, I'd appreciate it.

[nelsobe]

I have run this on artix7 and it works. However, needed to modify the test code since the part name is hard wired. Since this PR is based on a non-local branch, however, I cannot check that change in to the branch this PR is based on.

Suggest that either: (a) this PR be switched to be based on the local betterReverse2 branch (preferred since that branch also contains the artix7 device file generated from this PR code) or (b) the part name change and the updated device file be applied to Travis' repo so the travis-ci tests will complete so this can be approved and merged in.

[trharoldsen]

I updated the versioning numbers in RapidSmith to prevent old device files from being created. Unfortunately this means that the Artix7 part needs to be regenerated. Aslo, we need the kintexu part to also be regenerated.

[nelsobe]

Artix and Kintex parts are now generated. ReverseWireConnectionsTest modified to include tests for both devices. Both pass.

Device file generation process is more memory intensive than in the past. Previously, 11GB was sufficient for kintexu file generation. With the changes, 14GB was insufficient but 28GB was (so somewhere in that range).

Also xldrc file compression program was run on both xdlrc files. Compression ratio for artix file was 99%, 94%, and 29%. The resulting .cxdlrc file was ~53MB (as opposed to ~4.3GB) in size. Compression ratio for kintexu filewas 99%, 92%, and 55%. The resulting .cxdlrc file was ~226MB (as opposed to ~13.7GB) in size.

Need to decide where to store .cxdlrc files - no need to ever have to re-generate them unless Tincr generation process changes.

[DallonTG]

How long did it take to generate the device file with 28 GB? I know it was around 9 hours with 11 GB.

I don't really think it would be that useful to include the .cxdlrc files in the repository (not sure if you were considering that). If anything, I think it'd be more useful to just include the generated .dat device files.

Either way, if we want to include either the compressed xdlrc's or more device files, we could zip them all up and include them as part of each release. This way the actual repository wouldn't have to become excessively large due to additional files.

[nelsobe]

It took 11 minutes when run with 28GB of memory. So, it was clearly memory bound.

I agree – even the compressed xdlrc files are too large for the repo. We will want to decide where we might keep them so they are available.

From: Dallon notifications@github.com Reply-To: byuccl/RapidSmith2 reply@reply.github.com Date: Thursday, January 18, 2018 at 4:14 PM To: byuccl/RapidSmith2 RapidSmith2@noreply.github.com Cc: nelsobe nelson@ee.byu.edu, Comment comment@noreply.github.com Subject: Re: [byuccl/RapidSmith2] Improved XDLRC parsing and much faster device generation (#326)

How long did it take to generate the device file with 28 GB? I know it was around 9 hours with 11 GB.

I don't really think it would be that useful to include the .cxdlrc files in the repository (not sure if you were considering that). If anything, I think it'd be more useful to just include the generated .dat device files.

Either way, if we want to include either the compressed xdlrc's or more device files, we could zip them all up and include them as part of each release. This way the actual repository wouldn't have to become excessively large due to additional files.

— You are receiving this because you commented. Reply to this email directly, view it on GitHubhttps://github.com/byuccl/RapidSmith2/pull/326#issuecomment-358813953, or mute the threadhttps://github.com/notifications/unsubscribe-auth/APjxbjZjh1oDCQiUA1oAZkEFAxNu595eks5tL9BDgaJpZM4RVf5Y.

---

[http://home.byu.edu/home/sites/default/files/mono1web.jpg]

---

[trharoldsen]

Ready to merge. Should probably be targetted as a 2.0 release as the device files are incompatible.

[DallonTG]

Accepted and merged in #340