search

puhep / pudb

Purdue CMS FPix Database
0 stars 0 forks source link

Generate POS config Files #192

Closed drberry85 closed 7 years ago

drberry85 commented 8 years ago

Hey Greg,

I've already sent you a detailed email on how to do this. It may be best to just get it working for one module, and then add a button to do it for all of them. How you implement it is up to you. But keep in mind that we don't want to push a button 672 times to get the configs for 672 modules.

Doug

gneeser commented 8 years ago

Doug, I've gotten started on the project and it's now my top priority. I'll keep you posted as soon as I have something worth posting.

-Greg

gneeser commented 8 years ago

Doug, I've made some progress getting towards generating the config files, but the issue I wanted to make you aware of is that not all modules have config files, including some modules that have been mounted. This is because we implemented the upload capability late in March, but by then there had been a number of modules that had already passed testing. For example, the module you mentioned in your email, M-B-7-10, was tested in mid-March, so its extra files folder is empty.

I'm currently working on a side project to get you a list of modules that don't have config files.

-Greg

drberry85 commented 8 years ago

Greg,

Yes, we'll have to re-upload results that are missing config files for modules that are mounted on blades. This should be fairly easy to do.

Doug

On Tue, Jul 26, 2016 at 11:26 AM, gneeser notifications@github.com wrote:

Doug, I've made some progress getting towards generating the config files, but the issue I wanted to make you aware of is that not all modules have config files, including some modules that have been mounted. This is because we implemented the upload capability late in March, but by then there had been a number of modules that had already passed testing. For example, the module you mentioned in your email, M-B-7-10, was tested in mid-March, so its extra files folder is empty.

I'm currently working on a side project to get you a list of modules that don't have config files.

-Greg

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/puhep/pudb/issues/192#issuecomment-235323742, or mute the thread https://github.com/notifications/unsubscribe-auth/AElNLO-R2RQMagVucWo4UmCqBkih8oJ6ks5qZjUzgaJpZM4JJx9T .

gneeser commented 8 years ago

Doug, I added a filter on the Tested Modules List page that lets you filter by whether or not the module has config files. There are only four modules that have miscellaneous files but not the full/proper configs: M-I-3-41, M-J-4-27, M-J-4-13, M-J-2-19

I'll get back to work on generating the POS config files.

-Greg

drberry85 commented 8 years ago

Greg,

Only 4 modules! This is great! It should be very easy to get those configs in the DB.

Doug

On Tue, Jul 26, 2016 at 1:08 PM, gneeser notifications@github.com wrote:

Doug, I added a filter on the Tested Modules List page that lets you filter by whether or not the module has config files. There are only four modules that have miscellaneous files but not the full/proper configs: M-I-3-41, M-J-4-27, M-J-4-13, M-J-2-19

I'll get back to work on generating the POS config files.

-Greg

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/puhep/pudb/issues/192#issuecomment-235354695, or mute the thread https://github.com/notifications/unsubscribe-auth/AElNLPIIu4n5xh4Jh2G9PfwI4HsTW4L5ks5qZk0UgaJpZM4JJx9T .

NickHinton commented 8 years ago

I hate to be the bearer of bad news, but there are only 4 modules that have extra files that are NOT config files.

There are 311 (!) modules that do not have config files. 194 of those modules have been rejected, and I would assume you don't care about those. That leaves 117 that still need config files.

drberry85 commented 8 years ago

Well that's not great...

gneeser commented 8 years ago

Doug, I'm still working on the name mapping, but other than that I think I have the config files properly generated for one module. Could you take a look and see if there's anything horrifically wrong?

http://inky.physics.purdue.edu/cmsfpix//Submission_t/download/POS_config_files.php?name=M-Y-H-NP

-Greg

drberry85 commented 8 years ago

Greg,

It looks like you're using the untrimmed dac parameters file. So either we're uploading the wrong file, or you're using the wrong one. You should be using a file the has a 35 in the name (e.g. dacParameters35_C0.dat). This has the calibrated dac values. The other file (e.g. dacParameters_C0.dat) has the values after the pretest but before the calibration. If you don't have a dacParameters35 file that has a 35 on the end, then we need to re-upload the results with the proper configs.

Also the rules for the masking have changed as we may need to install modules that will have a single bad DC masked. So we can no long assume that all the pixels will be unmasked. To figure out what pixels need to be masked you need to read in the defaultMaskFile.dat line by line. You can seen an example of that here https://github.com/fnal/FPIXUtils/blob/master/DatToPOS.py#L130. Basically, there are 5 cases.

  1. File is empty, unmask everything
  2. Line is 'roc x'. Mask all the pixels on ROC X.
  3. Line is 'col x y'. Mask column Y on ROC X.
  4. Line is 'row x y'. Mask row Y on ROC X.
  5. Line is 'pix x y z'. Mask a single pixel on ROC X, in column Y, and row Z

Let me know if you have questions, Doug

On Mon, Aug 1, 2016 at 1:44 PM, gneeser notifications@github.com wrote:

Doug, I'm still working on the name mapping, but other than that I think I have the config files properly generated for one module. Could you take a look and if there's anything horrifically wrong?

http://inky.physics.purdue.edu/cmsfpix//Submission_t/download/POS_config_files.php?name=M-Y-H-NP

-Greg

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/puhep/pudb/issues/192#issuecomment-236669781, or mute the thread https://github.com/notifications/unsubscribe-auth/AElNLASLoZ4UnzwxE2wKkT9z6hzpD5FNks5qbj6TgaJpZM4JJx9T .

gneeser commented 8 years ago

Doug, As for the trimmed DAC files, I initially looked for them, but they seem to be missing for at least a few modules, including the one I used as my test case.

I'll keep working on the rest and keep you posted.

-Greg

gneeser commented 8 years ago

Doug, Looks like I have some bad news. I ran a search for modules that did have the trimmed dac files, and I didn't find any at all. You can see on the following page that there are trimmed files for the other categories, but not the dac files: http://inky.physics.purdue.edu/cmsfpix//Submission_t/download/configfiles.php?name=M-Y-H-NP

This seems to be the case for all modules. That suggests to me that we would have to re-upload config files for every module we're interested in.

-Greg

drberry85 commented 8 years ago

bummer

On Thu, Aug 4, 2016 at 1:50 PM, gneeser notifications@github.com wrote:

Doug, Looks like I have some bad news. I ran a search for modules that did have the trimmed dac files, and I didn't find any at all. You can see on the following page that there are trimmed files for the other categories, but not the dac files: http://inky.physics.purdue.edu/cmsfpix//Submission_t/ download/configfiles.php?name=M-Y-H-NP

This seems to be the case for all modules. That suggests to me that we would have to re-upload config files for every module we're interested in.

-Greg

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/puhep/pudb/issues/192#issuecomment-237647689, or mute the thread https://github.com/notifications/unsubscribe-auth/AElNLHYZUjBAxOO6A28lysjsHLgHQzs4ks5qcjSCgaJpZM4JJx9T .

drberry85 commented 8 years ago

Greg,

I just submitted the correct dac files fox this module. Try generating the configs again and I can compare. I am working on a solution to get this fix in for all modules.

Doug

gneeser commented 8 years ago

Doug, How does it look now?

drberry85 commented 8 years ago

Greg,

They look perfect! I'll get on updating all the .dat files for all the modules. I can generate them, but I'm working on a way to upload 800+ zip files in an automated way so I don't have to make the undergrads do it.

Doug

gneeser commented 7 years ago

Doug, I've added a couple things. First off, I added the "official name of position" to the database for all modules that had it populated in the spreadsheet you sent me - though I haven't quite worked out how to automate it for the rest. So that gives us 336 modules with the new position name, which is displayed on their Full Test Summary pages.

Also, for each of these modules, I made the POS config files and added a button (also on the Full Test Summary pages) to access them. I'm currently working on making the list of modules with POS config files searchable via the Tested Modules List page.

Is there another way that we should be able to access these files? As of now, the only way is to go to each module and look at its POS configs page (in the same way you access the "Extra Files" feature).

-Greg

gneeser commented 7 years ago

Just as a quick reference, http://inky.physics.purdue.edu/cmsfpix//Submission_p/summary/summaryFull.php?name=M-Y-H-NP

This module is one of the 336 with all the features I talked about.

gneeser commented 7 years ago

I just pushed out an update to the Tested Modules List page to search by the new criteria.

drberry85 commented 7 years ago

Greg,

I checked out a couple of modules and they look good. The last step now is to add a button to the main page that downloads a tarball, which contains all the configs. The files for each module should be in a different directory, and it should only have files for modules that are mounted on blades.

Let me know if you have questions or problems, Doug

On Thu, Sep 1, 2016 at 7:12 PM, gneeser notifications@github.com wrote:

I just pushed out an update to the Tested Modules List page to search by the new criteria.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/puhep/pudb/issues/192#issuecomment-244147019, or mute the thread https://github.com/notifications/unsubscribe-auth/AElNLOeXgBTod6n-2NdAvhnZj7SZiThTks5qlweIgaJpZM4JJx9T .

drberry85 commented 7 years ago

Greg,

Is there another way that we should be able to access these files? Yes, Jordan is not going to want to download 2688 separate files if needs to get the original pixel configs for some reason. So that's why we need the download tarball option. As I said, this tarball needs to contain all the POS config files for modules that are mounted on the disk.

I'll send you another spread sheet when we're done building the next HC. The reason you can't generate the names is because you don't know what disk name coordinates with what HC position and disk number (e.g. is MO6 is BmI_D3), but BpI and BmO haven't been build yet, so it's impossible to know which disk goes where.

Doug

On Thu, Sep 1, 2016 at 6:23 PM, gneeser notifications@github.com wrote:

Doug, I've added a couple things. First off, I added the "official name of position" to the database for all modules that had it populated in the spreadsheet you sent me - though I haven't quite worked out how to automate it for the rest. So that gives us 336 modules with the new position name, which is displayed on their Full Test Summary pages.

Also, for each of these modules, I made the POS config files and added a button (also on the Full Test Summary pages) to access them. I'm currently working on making the list of modules with POS config files searchable via the Tested Modules List page.

Is there another way that we should be able to access these files? As of now, the only way is to go to each module and look at its POS configs page (in the same way you access the "Extra Files" feature).

-Greg

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/puhep/pudb/issues/192#issuecomment-244133340, or mute the thread https://github.com/notifications/unsubscribe-auth/AElNLMpxr0tQXLdnImQZyAA9E75UPqisks5qlvv8gaJpZM4JJx9T .

gneeser commented 7 years ago

Doug, Check this page out: http://inky.physics.purdue.edu/cmsfpix//Submission_t/summary/POS_config_download.php?

Does that get you everything that you need? All I did was tar the entire POS_configs directory (which was already structured with sub-directories based on the official position name). I wasn't able to test it because I don't have the software on this PC.

If it checks out I'll push that page to the production area to make it more accessible.

drberry85 commented 7 years ago

Greg,

I did a few stop checks and it looks good. I do have one question. Do you know why there is a directory for both FPix_BmI_D3_BLD16_PNL1_RNG2 and M-Y-H-NP?

Doug

On Tue, Sep 6, 2016 at 12:27 PM, gneeser notifications@github.com wrote:

Doug, Check this page out: http://inky.physics.purdue.edu/cmsfpix//Submission_t/ summary/POS_config_download.php?

Does that get you everything that you need? All I did was tar the entire POS_configs directory (which was already structured with sub-directories based on the official position name). I wasn't able to test it because I don't have the software on this PC.

If it checks out I'll push that page to the production area to make it more accessible.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/puhep/pudb/issues/192#issuecomment-245025548, or mute the thread https://github.com/notifications/unsubscribe-auth/AElNLPqghnfPPbGDN-4W81B5AAjadkFcks5qnaKWgaJpZM4JJx9T .

gneeser commented 7 years ago

Ah, it looks like I forgot to remove that directory, so it got caught up in the tar. That directory is from back before I generated the configs using the official name of the position.

Try downloading it again - that directory should be removed.

gneeser commented 7 years ago

Any other items on the to-do list for this issue?

-Greg

drberry85 commented 7 years ago

I don't think so. If Jordan has issues, we may reopen it.

On Tue, Sep 13, 2016 at 2:33 PM, gneeser notifications@github.com wrote:

Any other items on the to-do list for this issue?

-Greg

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/puhep/pudb/issues/192#issuecomment-246797376, or mute the thread https://github.com/notifications/unsubscribe-auth/AElNLIfeTcNh0XbXjK6KqYXic4tHjvqXks5qpvqIgaJpZM4JJx9T .

drberry85 commented 7 years ago

Hey Greg,

The link on the DB for generating the configs is busted: http://inky.physics.purdue.edu/cmsfpix//Submission_p/summary/POS_config_download.php

Doug

gneeser commented 7 years ago

Check it now, should be working again.