Layout Database for CHIME Outriggers

[pboyle-mcgill]

• Layout Database Overview https://arxiv.org/pdf/1410.8418 Layout_Database_ARXIV_1410.8418.pdf

• Layout Database Interface: https://bao.chimenet.ca/layout/component.php

• Layout Database file: https://github.com/chime-experiment/ch_util/blob/master/ch_util/layout.py

• Layout Database Naming Convention: https://docs.google.com/spreadsheets/d/1Sn8_ny-i6sJFZybJgQROhPzNQ5hW-YBSkyjylhv3q8A/edit#gid=1807920613

• Allenby Layout DB Issue: https://github.com/chime-experiment/allenby_infrastructure/issues/35

• GBO Layout DB Issue: https://github.com/chime-experiment/GBO_infrastructure/issues/24

Useful existing code bases:

• ch_util: https://github.com/chime-experiment/ch_util (some python code to interface with the layout db; see layout.py)
• ch_www: https://github.com/chime-experiment/ch_www (web php interface code)
• doc_assembly: https://github.com/chime-experiment/doc_assembly (Kiyo's code base use to scan and generate bar codes of components at the CHIME site). Entries are stored in an LTF file, and scanned using an android-based scanner.

Scanner that was used by Kiyo:

HONEYWELL, VOYAGER 1400G, USB SCANNER KIT, OMNI-DIRECTIONAL, 1D, BLACK, USB TYPE A 11.5M STRAIGHT CA (pandemic/legacy model) Kiyo paid $160 CAD, but $250 CAD is reasonable.

In doc_assembly:

• Code in field/: https://github.com/chime-experiment/doc_assembly/tree/master/field - Contains scripts that get run on the tablets in the field. There's different scripts for each field “task” (assembling a cassette, raising a cassette, etc). These produce "ltf" files.

• Code in script/: https://github.com/chime-experiment/doc_assembly/tree/master/scripts - Entries in the "ltf" files get added to the permanent layout database using these scripts.

• Code in single_use/: https://github.com/chime-experiment/doc_assembly/tree/master/single_use - Scripts for creating labels, which Kiyo printed on vinyl at kinkos. To generate the bar codes, the convention is the first 3 characters spec the type of component, after that sequentially. The conventions are quite loose.

• Code in testing_env/: https://github.com/chime-experiment/doc_assembly/tree/master/testing_env - Contains a script to duplicate the whole layout database to a local sqlite. Then you can fiddle with that database without worrying about breaking the main one. (VERY IMPORTANT!)

Suggestion from Kiyo: In the field, you break a USB to mini USB adapter about once an hour… so buy lots of them.

[pboyle-mcgill]

Running Order;

• [x] Aaron access to chime-experiment github.
• [x] Call meeting with Mandana to discuss how to get started, agree on naming convention, Allenby team member responsibility.
• [x] Aaron to call a meeting with Adam to discuss the ch_util interface and how it is used.
• [x] Aaron to present an overall plan on the ODWG.

Deliverables

• [ ] Overall Naming convention.
• [ ] Layout DB for three Outrigger sites.
• [x] Bug fixes for adding data, CHIME components ++.
• [x] Interface with ch_util and ODWG team.

Inputs

• [x] Python interface.
• [x] PHP Web interface.
• [x] Barcode scanning.

Purchasing:

• [x] Bar code scanner [e.g. HONEYWELL, VOYAGER 1400G, USB SCANNER KIT, OMNI-DIRECTIONAL, 1D, BLACK, USB TYPE A 11.5M STRAIGHT CA (pandemic/legacy model)]
• [x] Bar code printer [for printing mock bar codes on pieces of paper]

[pboyle-mcgill]

CHIME LNA Rev C Allenby Rev: C3 Quantity: 139 (1 of 140 was damaged in production) Date: Oct 2020 SN: LNA2522C to LNA2661C PCBs by Enigma Assembly: Dorigo

[aaronpearlman]

Update: New python backend for adding, modifying, and connecting components in the layout db is done.

In progress: Working on developing a local database (in python) that will be used to capture components at the site, which will later be uploaded to the layout db (with "connexions") using the new python backend that I've developed.

TODO: Barcode scanning not yet implemented. We'll implement that capability as a user-input protocol within the local database.

[aaronpearlman]

My current (on-going) implementation plan for the local db (in progress):

• [x] # TODO: Create initial database (use a dictionary, store in JSON file).
• [ ] # TODO: Each entry in the database should have all of the fields in the layout db (serial number, component, rev type, properties, availability, history, document, connexions).
• [ ] # TODO: Connexions in layout db are not directional, so we can store a list of ALL connexions for EACH component.
• [ ] # TODO: Permanent and non-permanent connexions are possible. Trivial to implement this capability (use a flag).
• [x] # TODO: Querying capability from the local database.
• [x] # TODO: Add/delete capabilities from the local database.
• [x] # TODO: Barcode scanning capabilities.
• [x] # TODO: Text-to-speech capability (e.g., to let the user verify that they scanned the correct part).
• [ ] # TODO: Compare to schema/layout design provided by Allenby team (implement logic, if desired, to avoid mistakes using scanner).
• [x] # TODO: Write database to a file (JSON), for storage.
• [ ] # TODO: Once database is filled out, then write a script to automatically populate the layout db by parsing the local db. We can modify components individually later, if there are updates, using the new python backend.

Comments & suggestions appreciated!

[aaronpearlman]

Question: Will the Allenby team provide a list of part numbers, component types, and revisions for their outrigger site? This information will be needed for the local database.

[mondana]

There is a list of components (e.g. LNA, FLA, cables, etc) that are barcoded and there are some that are not (e.g. vestibule cables). We can provide a list. There are also things that are "virtual". for example a focal-line cassette position. Some components are already entered in the database.

[mondana]

are you planning to replicate the database? I thought the decision (at some point in time?!?!) was to keep one global database.

[mondana]

As a requirement: there should be a way to replace components : sever connections, replace connection and able to provide a comment/date/user for the action. There is currently a web php interface in addition to the python interface. The web interface has been extremely useful during field work.

[aaronpearlman]

@mondana I have no plans to replicate the database. I intend to keep one global database. Everything will be propagated to the layout db that is currently in use, but we will have different telescopes (or graphs) for each outrigger site.

In case it's not clear, the purpose of the local database is to allow us to add components to the layout db when we assemble on site. We'll store these locally, and then run a pipeline that I'm building that will populate new entries into the global database.

The functionality to sever and/or replace connections and add user comments already exists (both in the web interface, and now in my python backend). You'll have this option also with the local database (e.g., in case we find a faulty cable, and want to replace it with a different part).

Any other issues/thoughts?

[mondana]

ah, I see. Not sure why we won't be adding them directly to the global database, what would the pipeline do? perhaps you can provide a use case or an example. Please also note that (I think!) we already have two graphs : Pathfinder and chime-site (I guess 26m is not a separate graph) . So we are going to have 3 more graphs: allenby, gbo, xxx,

[aaronpearlman]

The local database will give us more flexibility in several ways. For example, (i) it is easier to make changes since the data structure is essentially a simple python dictionary or look-up table, (ii) we won't need an internet connection nor will we have to worry about complicated library dependencies to make updates (as is currently needed for the layout db), and (iii) if we need to replace a part in the database, we won't have to worry about additional overhead (e.g., making sure we sever connections in order to replace a part in the middle of the chain). I'm extremely keen to proceed in this fashion. I believe this will minimize our work.

The pipeline will be run at the end. It will just add each entry (if it doesn't already exist in the global database), along with all of its properties, connexions, etc..., and populate the elements in the local db into the layout db. It will just loop over components and add them one at a time. The local db will serve as a temporary record of the connections we made on site.

Yes, we'll have 3 more geographic sites / graphs as you mentioned for each of the outriggers. The 26m also has a graph, so we'll ultimately have a total of 6.

[mondana]

At least two scanner/table setup, because at Allenby, we will be doing things in parallel and quite possibly need one at the bottom.

[mondana]

Niko informs me that we have the 3 scanners at chime site and can use at Allenby. Please note that at CHIME, we had to buy other scanners that worked in sunlight and dim lighting, had the multi-directional laser, and could tolerate certain skew. So I suggest we use those from CHIME, and buy one for Aaron's testing at McGill only.

[mondana]

@nikola1003 also expresses preference for using android phones instead, so you can use with LTE and upload on the spot. Remember the difference at Allenby site is that, we have LTE. At CHIME, we did not.

[nikola1003]

Or for that matter, LTE-enabled tablets (there is a bunch on the market so it should not be hard to source).

[aaronpearlman]

Keep in mind that we're building this for use at the other outrigger sites as well. At GBO, we won't have (and can't use) LTE, so uploading components to the permanent database in real-time is not part of our plan at the moment. We can discuss if you think this will be an issue.

Do you know the model of the tablets that were used in the past? Are they still available for use, or do we need to identify new ones?

[mondana]

Not sure if you are still planning to refer to the google sheet of conventions, I made some edits and can do more if that is the place to capture those kind of information. Note that components are manufactured and barcoded in their manufacturing series. So the range of serial numbers should be free form. The first 3 letters are just chosen as a convention, but in most cases the serial number NNNN can be anything depending on when the unit was manufactured. LNA999C or ADC557 for example. unless I am missing a point about that googlesheet.

[mondana]

Keep in mind that we're building this for use at the other outrigger sites as well. At GBO, we won't have (and can't use) LTE, so uploading components to the permanent database in real-time is not part of our plan at the moment. We can discuss if you think this will be an issue.

Note that although we should make it flexible enough to be applicable to other outrigger site, but I really think we should focus on getting Allenby to work. We may find later that signal chain and scanning strategies are different at those sites and they may not even care about tracking and scanning every component depending on the assembly procedure.

For example. There is one cylinder at Allenby while there are more in other sites (I think!)

[mondana]

I think this is the tablet.

[aaronpearlman]

Things should be pretty similar at the other outrigger sites, from my understanding. There will be one cylinder at all sites, and I expect scanning and logging components will be handled more-or-less identically.

There's also other issues with adding components to the permanent database in real-time, which we've discussed previously. For example, what happens if someone makes a mistake, or we need to replace something in the middle of the chain? All of this is much easier to handle if we just log all of the scans and connections, and then upload them to the permanent database at the end. We don't need internet or LTE for this, just some way to store the entries until we're done (i.e. the local database). It turns out that this is also how Kiyo and others did it in the past.

A separate protocol just for Allenby is a lot of work, which I'm not keen to do. Let's not over-engineer this, but instead design something that will be reusable and work at all of the outriggers!

[pboyle-mcgill]

TO DO:

1. Build a mock cassette.

   • [x] Bar code for a cassette.
   • [x] Bar code for a feed position (4) in total.
   • [x] Bar code for polarization 0 and polarization 1 at each feed position.
   • [x] Bar code for LNA to connect to each polarization (8 in total per cassette).
   • [x] Bar code for a short cable to connect to each LNA (8 in total per cassette).
   • [x] Bar code for a bulkhead connector to connect to each cable (8 in total per cassette).

2. Build a mock external bulkhead (say 4 x 4).

   • [ ] 4 x X-positions.
   • [ ] 4 x Y-positions.
   • [ ] read in the X-Y positions.
   • [ ] Connect a mock external cable (bar code) to a position.
   • [ ] Connect a mock internal cable (bar code) to a position.

3. Build a mock internal bulkhead (say 4 x 4).

   • [ ] 4 x X-positions.
   • [ ] 4 x Y-positions.
   • [ ] Scan mock internal cable (from # 2 above). This should tell you the X-Y position to do to.
   • [ ] Scan the X and Y position and cable to connect.

[pboyle-mcgill]

[mondana]

Just wondering what these photos are for?