code-423n4 / 2024-03-acala-findings

0 stars 0 forks source link

Analysis #51

Open c4-bot-10 opened 3 months ago

c4-bot-10 commented 3 months ago

See the markdown file with the details of this report here.

c4-pre-sort commented 2 months ago

DadeKuma marked the issue as sufficient quality report

DadeKuma commented 2 months ago
Warden has submitted HM AI Generated Report Useful Diagrams / Tables Insightful Content Useful Suggestions Format
✔️ ✔️ ✔️
c4-judge commented 2 months ago

OpenCoreCH marked the issue as grade-b

Bauchibred commented 2 months ago

Hi @OpenCoreCH, thanks for judging so fast, where as I understand the subjectivity around grading analysis/QA reports. I'd like to argue that this report should be reevaluated and given a higher grade, comparing this to the currently "selected for report" which we agree can be considered the best among the 2 current grade a reports, this report is very similar.

Not to make this comment long, here is the selected for report analysis submission's, table of content Serial No. Topic
01 Overview
02 Architecture Overview
03 Approach Taken in Evaluating Acala
04 Acala Modules Analysis
05 Call-trace Diagrams
06 Codebase Quality
07 Systematic Risks and Centralization

Here is the one present in this report


Overview in SFR covers the same the same thing as Brief Overview in this report

Architecture Overview and Acala Modules Analysis in SFR covers the same thing as Scope and Architecture Overview section and all three modules in this report, where this report even includes a way deeper analysis.

Approach Taken in Evaluating Acala in the SFR is covers the same thing as Approach in this report

Systematic Risks and Centralization in the SFR report only hints one risk, which in itself is Centralization risk, however this report list way more as can be seen in the Centralization Risks of this report

Considering the above, this report includes multiple insights on the potential Systemic Risks which the current SFR report does not have.

Some insights raised from the Codebase Quality section of the SFR report has been listed in this report, under the whole scope's Overview itself and some other improvements under Recommendations (which is not present in the SFR)

Whereas it's only going to be fair to note that the SFR report has an upperhand in the case of the attached diagrams, i.e the three Call-trace Diagrams section of the report, we assume this should be why it should have the SFR tag and have the upperhand.

However this report also includes the Testing Suite setups for all modules and other sections not present in the SFR report.

Considering Code4rena's judging criteria and the supreme court's verdict on analysis does not necessitate the attachment of diagrams/tables to Analysis, we believe it's unfair to mark this a grade B report as this is actually the only instance where the current SFR report has the upperhand over this.

DadeKuma commented 2 months ago

Hey @Bauchibred , I could provide you with some additional feedback as I did the first pass, but of course, the Judge has the final say.

1) Diagrams/Tables were not necessary to get a high-quality report, but I gave a few bonus points if they were present and useful. Note that this was one of the least impactful criteria. 2) The Analysis's structure is not important, as long as it's readable and conveys valuable information and insights, especially regarding systemic/technical/integration risks, architecture considerations, and centralization risks.

A few cons about this Analysis that influenced your score negatively:

1) Describing the project functions (which is just code documentation) is not useful to the Sponsor. Consider adding insights about the architecture itself, or specific mechanisms/"gotchas" that are not already covered by the docs/comments inside the codebase. 2) The Recommendations section was too generic (e.g., "having regular security audits" or "adding more tests" can be applied to ANY project; this is not useful to the Sponsor).