Expose fitting parameters

[andrea-pasquale]

After discussing with @Edoardo-Pedicillo we thought that it could be very useful to store for protocols also "raw" fitted parameters. For example if we try to fit a specific measurement with a function $f(x, p_1, p_2, \ldots, p_n)$ usually in Qibocal we identify some of the parameters as relevant and we store them in the runcard. If we store all parameters $p_i$ we could provide more reliable initial guesses and eventually spot shifts. This is interesting but the main concern will be where to store such parameters. Storing them into the platform it is not an option given that we will introduce a circular dependency between qibolab and qibocal given that qibocal is dictating which is the functional form for every fit. The only other option will be that qibocal should be able to store them in a separate file, which makes sense given that such file will only be used by qibocal and it will be not necessary at all to run circuits or to play pulses. Thoughts @Jacfomg @alecandido @hay-k @stavros11?

[alecandido]

Storing them into the platform it is not an option given that we will introduce a circular dependency between qibolab and qibocal given that qibocal is dictating which is the functional form for every fit. The only other option will be that qibocal should be able to store them in a separate file, which makes sense given that such file will only be used by qibocal and it will be not necessary at all to run circuits or to play pulses.

Agreed

After discussing with @Edoardo-Pedicillo we thought that it could be very useful to store for protocols also "raw" fitted parameters.

Aren't they already stored in the History? (and consequently in the report folder)

[andrea-pasquale]

Aren't they already stored in the History? (and consequently in the report folder)

Yes, they are. However, what I would like to have is some file that contains a serialization of those parameters that could then be loaded directly with the platform. Something like this:

"resonator_spectroscopy_high_power": [0.1, 0.2, ...]

For all protocols and all qubits.

[alecandido]

What if we improve the access to the report folder? Making a loader that is capable to retrieve that information. E.g. something like:

report = Report.load(path_to_report_folder)

report["resonator_spectrocopy_high_power"].parameters

I'm proposing this for two different reason:

1. avoid duplicating information that is already available
2. and it would be beneficial for other projects as well

Regarding 2., e.g., @GabrielePalazzo is parsing the folder on its side to load the parameters in the dashboard. Since it's already present, standardizing the access to this information would be great :)

[andrea-pasquale]

I see your point. However, my main concern is to launch a calibration starting from a previous calibration. Currently previous calibration means "serialized platform after running previous calibration". On top of that I want to have access to the previously found parameters.

[alecandido]

However, my main concern is to launch a calibration starting from a previous calibration. Currently previous calibration means "serialized platform after running previous calibration". On top of that I want to have access to the previously found parameters.

Agreed: the platform eventually should only contain the parameters required for the execution, while it's certain that Qibocal may be able to consume more information.

But I'm not sure I see why this information couldn't come from the report folder...

[andrea-pasquale]

But I'm not sure I see why this information couldn't come from the report folder...

It could come from the report folder. Something like

qq auto <runcard.yml> --previous <previous_report>

However, a report will contain only a limited number of protocols hence only a limited number of parameters...

[alecandido]

However, a report will contain only a limited number of protocols hence only a limited number of parameters...

Ok, so what you mean is that you'd like to accumulate a history of multiple calibrations, not just the last one. Isn't it?

(in my mind, if you have multiple routines to run, eventually they will be run in one shot... but I have to recall every time that this is not the case, or at least not yet)

In principle, it could even be:

qq auto <runcard.yml> --previous <report0> <report1> <report2> ...

and have an object that gives access to all of them (with logic like "lookup in the latest, if failing, fallback on the previous one, and so on").

However, the same could be achieved doing something like:

qq merge <report0> <report1> ...

to produce a new object (w/ or w/o plots and html) with all the information already merged.

According to what is simpler, instead of explicitly merging we could just maintain an index of calibrations (i.e. where to find the reports related to a specific platform, either locally or on the report server), and manipulate that (append new calibrations, remove outdated ones), and the Report.load could take an the index path as input.

[andrea-pasquale]

Combining report it is not feasible since we need with a good metric (the lookup is not enough in my opinion). After discussing with @stavros11 I know that there are plans to move some of the physics related object in qibolab like Qubit "somewhere else". I believe that the parameters that I want to save could be stored along side those values. Having a single file which is populated and then pass directly when qibocal is executed is the easiest solution for now.

[alecandido]

"Somewhere else" will be most likely in the reports...

[alecandido]

Let's say the following: during the execution a single object will be generated, the report. I would not proliferate on that, because "the report" is already a folder, so we can already put whatever inside (and even if it were a JSON file could contain as much as a folder).

We can decide if the extraction process for relevant parameters should happen on its own, with someone manually running it (the qq merge proposed above), or it will happen at lookup time (following an index, or just filtering somehow the available reports).

Eventually, the idea is to move everything to a database. But this would only be an improvement in performances and (hopefully) reducing custom handling code. We already have a database, i.e. the report storage on top of the file system. Not an optimal one, but we're not lacking features to implement this request. We just have to decide where further manual intervention is required (like deciding which reports should be taken into account, and even if individual parameters should be marked as reliable or not) and which will be the interface for handling that.

[andrea-pasquale]

Let's say the following: during the execution a single object will be generated, the report. I would not proliferate on that, because "the report" is already a folder, so we can already put whatever inside (and even if it were a JSON file could contain as much as a folder).

I agree with the fact in the report we could generate this file. However, we will generate an updated version of the file. For sure the configuration needs to be stored somewhere else. You could make the same argument for the parameters.json, which are instead stored in qibolab_platforms_qrc.

Eventually, the idea is to move everything to a database. But this would only be an improvement in performances and (hopefully) reducing custom handling code. We already have a database, i.e. the report storage on top of the file system. Not an optimal one, but we're not lacking features to implement this request. We just have to decide where further manual intervention is required (like deciding which reports should be taken into account, and even if individual parameters should be marked as reliable or not) and which will be the interface for handling that.

This is for sure the correct solution but it is a long term solution while I would like to do something quicker. Currently we could even say that qibolab_platform_qrc is our "database" so putting an extra file there shouldn't be that much of an issue.

[alecandido]

This is for sure the correct solution but it is a long term solution while I would like to do something quicker. Currently we could even say that qibolab_platform_qrc is our "database" so putting an extra file there shouldn't be that much of an issue.

Yes, exactly, that's the platforms database, with the content required for execution.

And then there is the other half of the database, with content that is not required for execution, but still an output of the calibration. And this is why I'm saying they are the reports.

I agree with the fact in the report we could generate this file. However, we will generate an updated version of the file.

Btw, at some point I'd stop generating an updated version of the file, and just generate the update (as it was initially intended...).

You could make the same argument for the parameters.json, which are instead stored in qibolab_platforms_qrc.

Exactly, that's why we are trying to remove the non-execution part of parameters.json out (removing Qubit's attributes unused in Qibolab). However, parameters.json is separate from everything else (and it would be separate even in whatever new layout) because it's the only part that is needed in absence of Qibocal. Everything else is meaningless, unless you're in the Qibocal scope.

If you like it more, you could rename the reports' folder qibocal_calibrations_qrc, and save it in Git (not sure if it's truly advantageous, since it will be almost fully auto-generated, but since it would be decoupled, I have nothing against that).

[andrea-pasquale]

Btw, at some point I'd stop generating an updated version of the file, and just generate the update (as it was initially intended...).

Fine, but if we generate the update some other file will need to be updated, right?

Exactly, that's why we are trying to remove the non-execution part of parameters.json out (removing Qubit's attributes unused in Qibolab).

Yes, but it is still there... It seems that every time a change is proposed it cannot be implemented, because in the (near most likely far) future something will happen. I know that all libraries are not super stable but we should try to do something to avoid this kind of situations, especially having in mind that qibocal should be the priority because we are planning to publish the paper soon.

Just to wrap up, I will start working on how to expose these parameters and next week with @scarrazza we could discuss exactly where to put this extra layer of information.

[alecandido]

Yes, but it is still there... It seems that every time a change is proposed it cannot be implemented, because in the (near most likely far) future something will happen.

Nope, you got me wrong. There is nothing you can't do. I'm just saying to do it in a way that is aligned with things that are going to happen, not to wait for those things to happen. And, to be fair, Qibocal has never been blocked by the development of 0.2 (I remember to have advised you in the first place to avoid basing Qibocal development on 0.2, until very close to completion).

Fine, but if we generate the update some other file will need to be updated, right?

This is currently happening manually in any case, because the files are being copied or even committed manually. We just need to write a convenient function/CLI (essentially a wrapper of a deep_update() for dictionaries) to use instead of cp.

I know that all libraries are not super stable but we should try to do something to avoid this kind of situations, especially having in mind that qibocal should be the priority because we are planning to publish the paper soon.

It is fine to prioritize the Qibocal development, even in terms of manpower, for the reason you say. However, that's definitely not a reason to sneak in new features at the price of maintainability. Qibocal will have to survive its paper.