Closed jbqubit closed 5 years ago
He does schematic review right now. It is very time-consuming activity because every single connectivity and component pin need to be verified. He already found over 20 hardware errors in SInara AMC. Documentation is not that critical right now. He will start it in January.
@jbqubit Thanks for inspiration documents, I think I'm familiar with them. As @gkasprow has written, these days I'm checking schematics. I will take me about 1-2 weeks. So January is good time to start.
Thank you for the update @filipswit. :)
@gkasprow said
@filipswit started seriously working on Sayma and Metlino documentation. Filip just finished schemtic review of all boards and found several issues. He will use tex templates from AFCK.
That's great! @filipswit Re-open this Issue (or create new ones) if there are questions.
In #129 @hartytp said
What do people think about the idea of using the Wiki as the main source of documentation? This could have a few advantages:
Easy for people to edit later on, for example to post measurement results or comment on hardware bugs we find
Allows us to keep the entire Sinara ecosystem documented in a single place, rather than having a tex file for each component
It's already a git repo, which makes version control easy
@hartytp I strongly disagree with this suggestion. A manual needs to be a well-structured, thought-out document that a new user can read through and get a clear picture of the piece of hardware in question, what different design decisions were made and why, or reference when questions come up. Once the hardware is built, it will be mostly static. This is totally different from a free-for-all discussion forum (where one might "post measurement results or comment on hardware bugs"). It is also very different from the distributed, non-hierarchical node structure of a wiki, with fuzzy boundaries on where the manual begins and ends. The manual can certainly be updated with things like lists of known bugs in the future (and should be!), but this should happen in the context of a structured document. The manual should live alongside the more fluid back-and-forth discussion you're talking about.
A manual needs to be a well-structured, thought-out document that a new user can read through and get a clear picture of the piece of hardware in question, what different design decisions were made and why, or reference when questions come up. Once the hardware is built, it will be mostly static.
This should all apply to the Wiki as well: the bulk of the content should be written during the initial documentation stage, and it should only undergo minor modifications to keep it up to date, or additions when new hardware is added to the ecosystem. When I talk about "measurement results" I mean significant pieces of characterisation, like phase noise spectra that all users want to see, not random things that people have done. I don't think a well-managed Wiki is ever a "free-for-all discussion forum" -- that's what the mailing list is for!
Would it make you less unhappy if I changed my suggestion to: make the documentation a GitHub project written in markdown (rather than Latex)? In that case, it's easy to integrate content from the manual into the Wiki. The major benefit of this as I see it is that we can have a single website which provides documentation for the entirety of Sinara (Metlino, Sayma, Kasli, DACs, etc). This is really helpful for new users who are trying to understand what the hardware ecosystem looks like and how everything fits together (it's much harder to have a stack of independent manuals that don't cross-reference properly).
NB we can restrict who has commit access to the Wiki so it's not a complete free-for all. Also, as the Wiki is a git hub project, we can also publish "releases" of it (e.g. after Sinara round 1), which freeze it -- or sections of it -- a certain points.
The major benefit of this as I see it is that we can have a single website which provides documentation for the entirety of Sinara (Metlino, Sayma, Kasli, DACs, etc).
I fail to see how this cannot be done with a single PDF, if you really think they're that entangled. Who said that there has to be one manual per board?
BTW, I don't agree that they are that entangled at the level of hardware. Each of these boards can operate on its own without the others.
The intended software and firmware makes them part of a larger ecosystem, but software documentation is clearly going to have to be elsewhere, i.e. part of the normal ARTIQ documentation.
I fail to see how this cannot be done with a single PDF, if you really think they're that entangled. Who said that there has to be one manual per board?
It can all be put into a PDF in principle. But, the plan is to start writing the documentation for Sayma and Metlino now, while the hardware ecosystem is still evolving rapidly. So, the documentation will either have to cover only those boards, and not cover the wider ecosystem (and how they interact with it), or run the risk of being obsoleted rapidly.
Why the preference for tex PDF over markdown files anyway? Markdown files can be saved and opened locally in a browser if desired (rather than over the web). They can also be complied into a PDF if that's important to you.
The intended software and firmware makes them part of a larger ecosystem, but software documentation is clearly going to have to be elsewhere, i.e. part of the normal ARTIQ documentation.
The way this hardware is used in an experiment, and the other hardware it plugs into makes it part of the wider ecosystem.
I fail to see how this cannot be done with a single PDF,
Also, if you use MD or similar then you can make the documentation easier to navigate by including sidebars etc.
Why the preference for tex PDF over markdown files anyway? Markdown files can be saved and opened locally in a browser if desired (rather than over the web). They can also be complied into a PDF if that's important to you.
If all we're talking about is document formatting platforms, I really don't want to argue about it. I refer to you to the many excellent references on how TeX is a wonderful document formatting system, produces attractive and easy to read documents, and why PDF is a very useful format. I was more talking about the nature and structure of a manual, not how it's rendered.
The way this hardware is used in an experiment, and the other hardware it plugs into makes it part of the wider ecosystem.
You are presuming a level of uniformity in how this hardware is used, which is both unfounded and unnecessary. How you use this hardware in your experiment, and what other hardware you plug it into, is on you to document. It does not belong in a Sayma manual. Sayma (and Metlino etc) are designed extremely flexibly, and can be used in all sorts of ways. I would leave the manuals separate.
My thinking here is that we can write the documentation for Sayma/Metlino as pages in md, so that they can be viewed as standalone web pages or even compiled into a PDF if desired. However, as they're md, it's also trivial to import them into the Wiki github project. The same pages/files can then be used as part of a general Sinara documentation Wiki, which covers the entire ecosystem.
You are presuming a level of uniformity in how this hardware is used, which is both unfounded and unnecessary.
Sure, there will be large differences in how we all use this hardware, but there is a lot of common ground too and it's helpful to make that clear. As things currently stand, the ecosystem will mainly be made of a few different main kinds of things (uTCA boards, Kasli + extensions, etc) which interface together in a small number of different ways (IDC/DRTIO/backplane). IMO, it's worth making that clear in the docs, rather than just documenting the functionality of each module independently without providing any context of why that functionality was provided, or what it is intended to be used for.
e.g. The Metlino VHDCI is designed to interface with the VHDCI carrier. So, it's nice to be able to follow a link from the Metlino docs directly to the VHDCI carrier docs. The VHDCI carrier is designed to interface with all of the extension modules (not just the TTL ones, but also the DACs, ADCs, camera link, etc) so it's useful to be able to directly link from the VHDCI carrier to a list of the available extension modules -- and it's useful for that list to grow as more modules become available. It's also useful to know that Kasli provides an alternate carrier for these same peripherals, and that it can interface with Metlino via DRTIO/fibre. It's also nice to have a high-level introduction to all of this in the beginning of the docs.
I agree with @hartytp that it's useful to include cross linking in the documentation. Since @filipswit and @gkasprow are writing the documentation I'm inclined to let them choose the language.
Some observations.
@jbqubit Works for me.
@jbqubit I agree. Along those lines, I suggest @filipswit writes a full manual to be delivered with the hardware (e.g. a TeX file or PDF) as the "official" documentation. We can then reformat it to an editable user version that can be posted and edited by the community.
Has anyone here used Pandoc (or similar) for converting between document markups? If so, does it work OK?
Okay, so to summarise what's been said (as I understand it), the plan is to write two forms of documentation:
WUT will write the board-level documentation for Sayma and Metlino as their "official" documentation effort. They will aim to do this in a way that makes it easy to integrate into the overall Sinara documentation.
Questions:
@jordens How does this fit in with your plans for documenting Kasli + the other hardware you're planning?
Anyway, we will write dedicated manuals/datasheets for each board. They are quite universal and can also serve other purposes.
Our referable way is latex because we can nicely publish vector images but I’m open to other forms.
I also think that board manuals should be separate documents with all details which are needed to fully explore their possibilities.
Such things like assembly options, diagnostics, testpoints are not very interesting for ARTIQ user. And for such document we need good tool that supports vector graphics, search functions and looks more like manual then webpage.
This is example of such documentation
http://creotech.pl/wp-content/uploads/2015/08/CTI-AFCK_datasheet.pdf
There should be another general document covering certain use cases that embeds interactive links to detailed documentation and here wiki would be better idea since the content will develop intensively while board manuals will be static.
@hartytp it seems we came to the same conclusion:)
Pandoc works well.
@hartytp Right now I am just writing an overview about Kasli. I'll do so in reST. Bu I am much more concerned with the content that with the format.
@jordens: right now I am just writing an overview about Kasli.
Great! If you can cover everything we need to make Kasli compatible extensions (e.g. plans for a Kasli BP if there will be one, etc) in the process, then that will be a really useful resource.
Bu I am much more concerned with the content that with the format.
As am I. Anything that integrates well into a nice, easily navigable webpage works well for me.
As I understood: @gkasprow , @jasonamini , @hartytp , WUT prepare 3 separate docs (Sayma AMC, Sayma RTM, Metlino).
I prefer to write it in a TeX and export it to PDF, cause I've a good template. For now I'm working on block scheme, I2C map and some specifications of components of Sayma AMC.
@filipswit How's the documentation progressing? Please give an update.
Generally I have cores for all documents. Some pictures are just placeholders for appropriate pics. Signal table for Metlino is incorrect. SaymaRTM.pdf SaymaAMC.pdf Metlino.pdf
Please say what else would you like to see in doc.
Thanks @filipswit! Some comments follow.
Xilinx Kintex UltraScale
MGT
Figure 4: General Block Scheme
FPGA bootstrapping
FMC
SFP
uTCA.4 compliance
USB-UART
power
JTAG
Signal Tables
[ ] Get additional narrative about Project Description from the wiki.
[ ] Cite high speed ADC and DAC ICs by name and summarize high-level specifications -- that's what this board is about!
Figure 4 Block Scheme
AFE mezzanine
clock distribution mezzanine
housekeeping signals
clock distribution
FPGA bootstrapping
Figure 5
See comments for Sayma_AMC. Most apply here.
@jbqubit, @dhslichter Many Thanks for your comments!! It is much easier to write sth. with such a good feedback.
Now that I have an Sayma_AMC to play with I can see additional documentation needs.
JTAG
Front panel
list of headers on PCB
SW1
I2C mux
RGMII
interface between AMC FPGA and MCU uP
@filipswit What's the status of the documentation? The hardware has shipped.
Expect documentation update from @filipswit https://github.com/filipswit witnin 2..3 days. He is working hard on it.
@jbqubit Here you have some update of documentation. I will add here every few days some newer version.
It would be best to hold off on posting until you've generated a version you consider ready for feedback. In particular, please implement (or reject) all the suggestions I made for the previous version before reposting. That makes it easier for all the contributors to comment/discuss the same document.
Feedback on what you posted yesterday (Issue Date November 29, 2017).
p. 4 -- lots of the acronyms don't make sense without a short introduction to uTCA.4 p. 4 -- transceiver type, maximum data rate p. 4 -- spell out what MMC is an abbreviation for, give brief synopsis of properties of LPC1776 p. 7 -- back view of PCB connectors p. 7 -- consider illustrating the location and description of the different IC's in the way that iFixIt does p. 13 -- LED call outs look great! p. 18 -- excerpts from schematics are helpful at explaining; more verbose would help
Suggestion: Ask a friend who knows nothing about Sayma to read your documentation and ask questions. This will be a type of feedback that is invaluable as it is from the perspective of a person who will use the documentation to understand what the board does.
@filipswit you can add introduction to MTCA - just copy part of my article about it.
p. 4 -- lots of the acronyms don't make sense without a short introduction to uTCA.4
Would you like an introduction or a glossary?
Do both.
@filipswit How's the documentation coming? Your docs are helpful for interfacing with M-Labs too. :) https://github.com/m-labs/artiq/issues/854#issuecomment-349905800
@filipswit ping
This is what I've done so far.
p. 4 -- lots of the acronyms don't make sense without a short introduction to uTCA.4
There is a glossary, please say which one should I add yet. Introduction will come later.
p. 4 -- spell out what MMC is an abbreviation for, give brief synopsis of properties of LPC1776
abbreviation is in glossary. Properties will be later.
p. 7 -- back view of PCB connectors
will take a photo this week
p. 7 -- consider illustrating the location and description of the different IC's in the way that iFixIt does
p.17 - p18 Did you expect sth like this?
p. 13 -- LED call outs look great!
Thanks!
p. 18 -- excerpts from schematics are helpful at explaining; more verbose would help
Text in progress
Did you change the max size of attached file? I can barley fit with my doc in 10MB, had to cropp some ilustrations.
Thanks @filipswit -- not enough changes to warrant a re-read. Please respond to existing comments before sending for review. See eg https://github.com/m-labs/sinara/issues/133#issuecomment-346499637 Also please post both source and PDF next round.
@filipswit When you've implemented documentation changes in this Issue please close it. For next documentation review please create a new Issue for that revision. This is Issue is getting too long.
@filipswit How's the documentation coming? Please feel free to check items off (above) as you implement them.
@jbqubit I've checked few boxes before in this topic: https://github.com/m-labs/sinara/issues/430 but from some reason I can't do it now. I can check it off but it's not saved. I it is the same above.
@filipswit It's getting critical that the documentation be finalized for Sayma v1. The PLL and Ethernet Issues will be soon resolved. Docs are one of the last parts of the WUT contract.
@filipswit @gkasprow I'm not confident that details such as the following are being added to the Sayma v1.0 documentation.
I added most of the Issues, some of them may still nedds some corrections. But I think is good moment to get some new remarks;)
Actualy, I dont't know what you expect about this "document which ICs have Ethernet access for Sayma 1.0. m-labs/artiq#854" - for now just added placeholder.
The source code is already in git. SaymaAMC.pdf
Awesome @filipswit! Looking forward to jumping in and reading. :)
For reference, this version of the documentation provided by @filipswit has commit id https://github.com/m-labs/sinara/commit/e745783298d77aebc46f4e7b66e0e10ec732c286.
This is a big job @filipswit! Thanks for tackling it. Some comments are below.
Start with a section outlining the ARTIQ Sinara ecosystem. Language from wiki seems fine. https://github.com/m-labs/sinara/wiki/images/overview.png
Sinara is an open-source hardware ecosystem originally designed for use in quantum physics experiments running the ARTIQ control software. It is licensed under CERN OHL v1.2.
Control electronics used in many trapped-ion and other quantum physics experiments suffers from a number of problems. In general, an ad-hoc solution is hastily put together in-house without enough consideration about good design, reproducibility, testing and documentation. This makes those systems unreliable, fragile, and difficult to use and maintain. It also duplicates work in different laboratories. In addition, the performance and features of the existing systems (e.g. regarding pulse shaping abilities) is becoming insufficient for some experiments.
To alleviate those problems, Sinara aims to be:
Sinara is currently developed by a collaboration including M-Labs, Warsaw University of Technology (WUT), US Army Research Laboratory (ARL), the University of Oxford, the University of Maryland and NIST. The majority of the hardware was designed by WUT. The work was funded by ARL, Duke University, the University of Oxford, and the University of Freiburg.
Following the ARTIQ model, an experiment consists of a Core Device (master) -- typically either a Metlino or Kasli -- controlling multiple slave devices in real time using ARTIQ's distributed real-time IO (DRTIO) protocol. DRTIO provides both gigabit communication links and time distribution over copper cable or optical fibres. It synchronises all device clocks, ensuring they have deterministic phase relationships, and enables nanosecond timing resolution for input and output events across all devices in the experiment. More detailed information about communication between devices and time distribution inside Sinara can be found here.
Sinara uses two main form factors for hardware requiring real-time control: microTCA (uTCA) and Eurocard Extension Modules (EEM). Non real-time hardware is typically connected to the host PC using Ethernet.
MicroTCA (uTCA) is Sinara's preferred form factor for high performance hardware with high-speed data converters requiring deterministic phase control, such as the Sayma Smart Arbitrary Waveform Generator (SAWG). Information about uTCA hardware, including a list of parts needed to build a Sinara uTCA crate can be found here.
EEMs provide a lower cost, simpler platform than uTCA for hardware that requires real-time control, but not bandwidth or complexity of uTCA hardware.
Extension modules connect to a carrier, such as Kasli or the VHDCI carrier, which provides power and DRTIO. They are designed to be mounted either in stand-alone enclosures, or in a rack with a carrier, and connect to the carrier via ribbon cable. More details about the extension module standard can be found here.
uTCA hardware interfaces with the extension modules either directly, using a VHDCI carrier, or indirectly, using a Kasli DRTIO slave.
Rename section to "uTCA.4 Overview".
"We make use of the most recent extension to the uTCA standard, uTCA.4" is not accurate. True we use uTCA.4. But latest is uTCA.4.1 is the latest.
Figure 1: There's an extra board in AMC 7 that's fine to have in the picture but confusing if the boards are not labeled. Add colored overlay labeling MCH, Sayma AMCs, power module.
New Figure: add photo of Sayma_AMC attached to Sayma_RTM sitting on bench so how they're connected is clear.
Just lump this together with pervious section. Yes, please add data sheet.
Delete this section since it's covered in the new Sinara overview.
Here's what I ordered in the past.
Add additional content.
Please adjust language. Sayma is distinct from SAWG. Sayma is hardware that supports M-Lab's Smart Arbitrary Waveform Generator (SAWG) gateware.
Combine Section 6 and 8. No need to list hardware components twice.
Link back to git repository for hardware versions.
Analog Front End acronym is AFE.
Specification here. Link is broken.
~~ Flexible feedback to SAWG parameters planned. Specification here. ~~
~~ – motivation for this FPGA choice is here ~~
data rate is 1.2 GS/s at 16-bit
data rate is up to 1.2 GS/s at 16-bit
For transceiver usage make a table for each Transceiver (Rx and Tx) listing where its routed. Some transceivers are routed to multiple places (eg SATA and backplane fabric). I would leave out ARTIQ-dependent detail (eg DRTIO-upstream).
Expand glossary... MCH, AMC, COTS, HEPP, RTM, RFBP, AFE, Sayma, Sinara, BaseMod, MixMod, EEM, uTCA, MMC,
Move glossary to very beginning/end. It's funny floating in middle of manual.
Merge with Section 6. Add subsections for AMC and RTM.
Lots of typos in preample.
This is an annoyingly partial-information schematic. I propose changing caption to "Sayma AMC external I/O" and just make it about that. Add other components only as needed to explain IO routing (eg FPGA, MMC, UART).
Here and in all similar following Figures...
The clock information here is too spare to be useful. Remove Si5324 and tell reader to refer to Figure 10 for clock domains.
Add symbol for front panel Clock SMA. Draw arrow to floating text "see clock Figure 10".
I/O--> TTL Use name that matches schematics.
USB, SMA, SMA, SMA are physically on edge of board. Give them a different color box than yellow and put them on the front edge of the board in the drawing.
Give SMA same names as in Schematic.
Strike uFL and MSATA as these are internal-facing connectors.
USB-->UART Draw arrow to floating text "see USB interface in Fig XX".
Caption should indicate that you're illustrating transceiver routing. Seems like there's room on this figure to also include Ethernet routing (and it's corresponding SATA).
Use two distinct grey wires for each of two FPGA to AMC+SATA.
Fonts "1x" etc are too small.
Indicate in caption that hex code is I2C address. Put hex code in parenthesis in all cases.
The grey wires in Fig 8 are very confusing. This is exacerbated by lack of narrative in Figure caption. Explain why of two MUXs.
There are several I2C busses. Distinguish between them and what's attached to each. Give busses names so they can be referenced in other parts of text. You use some bus names in 19.1 eg I2C_SENS and PMI2C. Be consistent (I2C vs _I2C).
Discuss I2C bus sharing between MMC and FPGA. How does that work?
Circle indicating that grey lines are joined should be in the front-most layer of the drawing package.
Say what the different LM75 are attached to in Sensors block.
Put J27 on front side of the board schematic.
Discuss in caption role of different oscillators and what the point of the clock recovery chip is. See following writeup.
https://github.com/m-labs/sinara/wiki/SinaraClocking
No need to mention DRTIO as that's an ARTIQ feature.
What clock should I apply if I want to operate Sayma? What power level?
Try to put these on the same page as the call out table.
Pair of RTM connectors and AMC card-edge connector labels are missing.
Try to put Figure 13 on the same page as the LED table. Move IC column to third column.
Doesn't the "nominal state" of the MMC involve some of the LEDs flashing?
Failure column doesn't seem useful.
Somehow the LED table ends up being difficult to read. I know I asked for "nominal sate column" but didn't help. For each LED I want to know what IC drives it an the relationship between LED state and IC state. Examples,
Descriptors in call out table are not altogether helpful. I want to know what IC is connected to each and where appropriate what net it's connected to and its function. Examples,
J15 and J26 are general purpose FPGA IO
J27 is clock output driven by Si5324 (CDR_CLK_CLEAN3)
XXX is FMC header -- why not call it J28??
PB3 is push button routed to MMC that does ___
USB to FT4232H IC that exposes 4 UARTs (See Section XX)
JTAG headers are connected to which chips?
That callout 6 points to two different LEDs is confusing. Why not just give them different callout numbers?
Narrative is needed to discuss what SW1 configures. What is is connected to? Master/Slave with respect to what bus?
Typos.
Consider moving this down to 14.2.
The interaction between MMC and FPGA is not well described.
Please say exactly what the expected behavior is for "Selection between MMC and USB is preformes automaticly."
What does this mean?? "When micro-USB is connected S siglnal is high and Multiplexer connects USB to FPGA." Which UART?
Consider moving this section up one level to 14.1 Say how UARTS are connected.
- UART0 is connected to X, configuration is 1N8 xxx baud
- UART1 is "
- UART2 is "
- UART3 is "
I don't see the end user use case for this: Another FTDI USB driver, the D2XX driver, can also be used with application software to directly access the FT4232H on the Mini Module though a DLL.
Tell user that there are two independent JTAG chains. Say what ICs are connected to each. Say what options a user has for driving each chain.
Move narrative about Scansta112 to before Figure 19. Figure 21 is super confusing. More explanation is needed in caption.
"JTAG can be connected either from USB (FT4232H) or from IDC header." Which IDC header? Which JTAG chain?
How does Sayma JTAG relate to uTCA JTAG?
This section isn't helpful. I'd just delete it.
Introductory orientation paragraph is missing. Outline.... As card is inserted... power is applied to MMC first. Explain that there are XXX power rails derived from +12V. There are several power chips ... VCCINT PSU, P5V0 PSU (helper PSU) and Exar LDOs. They are all are actively configured by MMC using an I2C bus, right. There's a particular power on order. What is it? Exar firmware monitors current and responds if its too high. Can MMC readout Exar debug information? How about VCCINT and P5V0 information? Is MMC supply information exposed to user?
Pull content from 17.2.3 into this section.
Figure 24 needs a caption. What's the point of this Figure? What are the different cells? What's the blue block? And OTP? UVLO? Please include narrative that describes the very nice power system you've built.
If there's a story to tell in Figure 25 additional narrative is needed. What's a group? Is there a power configuration file? Or is it embedded in the MMC firmware?
Showing a bunch of screen shots of Exar tool is lame documentation. What are you trying to say in Figure 26?
configures CPU, UART
sets port directions
configures I2C switch base address initializes I2C controller and chain (switch)
Which I2C busses?
checks if RTM is inserted, if yes, then enables its power, waits 200ms and initializes RTM power supply via I2C. It also configures Si5324 on RTM
runs task.
There's more detail needed here.
What version of LPCXpresso? Are there any particular configuration options? Say which chip to taret.
What version of NXP programmer software? Is NXP programmer a particular piece of hardware? If so which model number? Which header on PCB? Use designators from earlier.
How does this work? Why use USB vs header?
Narrative discussing uTCA housekeeping concept where MMC collects info and passes on to MCH via IPMI. Discuss that OpenMMC will run on MMC chip and that an OpenXXX will run on MCH and collect AMC device housekeeping information.
Discuss how I2C busses are shared between MMC and FPGA. Do this here or refer to earlier section where it's discussed.
temperature and current safety interlocks. Is this enforced by MMC or by Exar chips? How does a user know if an interlock is active?
This is being written by Creotech, right? Please ask that they commit their test jigs and code to repository.
Give this Appendix a name.
https://github.com/m-labs/sinara/wiki/Sayma-gateware#data-and-sample-rates
@filipswit Please refactor in light of my comments. I'd like to do review of AMC and RTM docs at the same time.
@filipswit is leading effort to write formal documentation of Sinara. When will the documentation effort start? For inspiration see documentation for AFC.