Solver details - Githubissues

max-radin commented 1 week ago

This PR makes a number of changes to the problem and solution schemas to allow for more details about the solver to be reported. See https://github.com/isi-usc-edu/qb-gsee-benchmark/issues/12 for more context.

Note also that the plan is to not include aggregated resources in the schema (e.g. total number of T gates across all tasks). Instead, resources should be provided on a per-task basis and any desired aggregation can be done in post-processing.

Changes to problem schema:

data_instance_objects renamed to tasks for more clarity.
Added task_uuid field to elements of tasks.

Changes to solution schema:

Added optional classical_resources, quantum_resources, and solution_details fields to elements of solution_data.
Renamed instance_data_object_uuid to task_uuid.
Renamed compute_details to solver_details for consistency.
Added solver_uuid to solver_details.
Moved compute_hardware_type into solver_details.
run_time no longer required for resource estimates in order to allow for reporting of logical resource estimates.
Solutions only required to report the overall runtime. Breakdown into preprocessing etc. is now optional.
Added optional classical_hardware_details, quantum_hardware_details, algorithm_details, and software_details fields to solver_details.

jtcantin commented 1 week ago

@.** We intend the |task_uuid| to be the same as the FCIDUMP uuid, partly to emphasize that the FCIDUMP uuid is the ground source of truth. Would this ease any backwards compatibility concerns?

On 2024-11-19 15:50, John Penuel wrote:

@.**** requested changes on this pull request.

On schemas/solution.schema.0.0.1.json https://github.com/isi-usc-edu/qb-gsee-benchmark/pull/14#discussion_r1849039169:

Please:

remove |short_name| from the top-level required fields

add |solver_short_name| as a required string field inside of the |solver_details| object. This is just a human-readable, possibly NOT unique field describing the solver. make it a required field.

any details on the |classical_compute_details| can be optional for now.

Currently I'm struggling with the premise that we have made some bespoke scripts and analysis that key off of FCIDUMP files specifically, not |task_uuid|. In general, I see the value of adding |task_uuid| to the |problem_instance| JSON.

Changing the |problem_instance| schema to include |task_uuid| will break some scripts and require updating and regression testing. I believe that our partners are also working on |problem_instance.json| files and they will have to update their JSON as well. Be sure to offer to help our partners update their |problem_instance| files.

— Reply to this email directly, view it on GitHub https://github.com/isi-usc-edu/qb-gsee-benchmark/pull/14#pullrequestreview-2446626298, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABC6NAKZCMZO6EUXAK4XS732BOQBXAVCNFSM6AAAAABSBAUCQSVHI2DSMVQWIX3LMV43YUDVNRWFEZLROVSXG5CSMV3GSZLXHMZDINBWGYZDMMRZHA. You are receiving this because you are subscribed to this thread.Message ID: @.***>

--------------fWZvAiduskHHnrYaPrA8ep6O Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: 8bit

<!DOCTYPE html>

***@***.***
We intend the task_uuid to be the same as the FCIDUMP uuid, partly to emphasize that the FCIDUMP uuid is the ground source of truth.
Would this ease any backwards compatibility concerns?

On 2024-11-19 15:50, John Penuel wrote:

***@***.*** requested changes on this pull request.

On schemas/solution.schema.0.0.1.json:

Please:

remove short_name from the top-level required fields

add solver_short_name as a required string field inside of the solver_details object. This is just a human-readable, possibly NOT unique field describing the solver. make it a required field.

any details on the classical_compute_details can be optional for now.

Currently I'm struggling with the premise that we have made some bespoke scripts and analysis that key off of FCIDUMP files specifically, not task_uuid. In general, I see the value of adding task_uuid to the problem_instance JSON.

Changing the problem_instance schema to include task_uuid will break some scripts and require updating and regression testing. I believe that our partners are also working on problem_instance.json files and they will have to update their JSON as well. Be sure to offer to help our partners update their problem_instance files.

—
Reply to this email directly, view it on GitHub, or unsubscribe.
You are receiving this because you are subscribed to this thread.

--------------fWZvAiduskHHnrYaPrA8ep6O--

jp7745 commented 1 week ago

task_uuid and instance_data_object_uuid should be separate UUIDs to key on for future flexibility. They should not be overloaded with the same value. New UUIDs should be generated for task_uuid when converting old-to-new problem_instance files. I hope that in the future we have more flexible inputs rather than just the FCIDUMP file format, but that's what we have at this time.

At this time, it's ok to update problem_instance.schema, but be sure to offer help to our partners to update to the new format during this Thursday's GSEE meeting.

jp7745 commented 1 week ago

To clarify: I'm ok with updating problem_instance.schema to include task_uuid. I'm asking for additional minor changes to the solution.schema like removing short_name and adding solver_short_name to the solver_details object before we merge.

Note also that the automatic JSON validation script will probably trash all of our problem_instance files. You'll have to pull them out of the /json_files_with_errors directory.

jtcantin commented 1 week ago

I've changed the short name related fields in the solution schema.

I believe the classical details fields (classical_resources and classical_hardware_details) and their subfields are all already optional in the solution schema, though let me know if I missed what you were referring to.

And, yes, I can help Matt/HRL with the task uuids.

On 2024-11-19 16:30, John Penuel wrote:

To clarify: I'm ok with updating |problem_instance.schema| to include |task_uuid|. I'm asking for additional /minor changes/ to the |solution.schema| like removing |short_name| and adding |solver_short_name| to the |solver_details| object before we merge.

Note also that the automatic JSON validation script will probably trash all of our |problem_instance| files. You'll have to pull them out of the |/json_files_with_errors| directory.

— Reply to this email directly, view it on GitHub https://github.com/isi-usc-edu/qb-gsee-benchmark/pull/14#issuecomment-2486791332, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABC6NAILQCO5UFQC3CR3VED2BOUV7AVCNFSM6AAAAABSBAUCQSVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDIOBWG44TCMZTGI. You are receiving this because you commented.Message ID: @.***>

--------------mgUGMfnofq2wSABo0FcnMU0e Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: 8bit

<!DOCTYPE html>

I've changed the short name related fields in the solution schema.

I believe the classical details fields (classical_resources and classical_hardware_details) and their subfields are all already optional in the solution schema, though let me know if I missed what you were referring to.

And, yes, I can help Matt/HRL with the task uuids.

On 2024-11-19 16:30, John Penuel wrote:

To clarify: I'm ok with updating problem_instance.schema to include task_uuid. I'm asking for additional minor changes to the solution.schema like removing short_name and adding solver_short_name to the solver_details object before we merge.

Note also that the automatic JSON validation script will probably trash all of our problem_instance files. You'll have to pull them out of the /json_files_with_errors directory.

—
Reply to this email directly, view it on GitHub, or unsubscribe.
You are receiving this because you commented.

--------------mgUGMfnofq2wSABo0FcnMU0e--

isi-usc-edu / qb-gsee-benchmark

Solver details #14