Unclear +/- sign of some facts

Hi,

thank you so much for the work you have put into this library.

I am using this library to process SEC XBRL filings to fetch information from P&L, Balance Sheet and Cash Flow Statements, etc.

A couple of challenges I have run into:

Positive/ negative sign of facts is not always clear I.e. for AAPL 2021-09, IncreaseDecreaseInAccountsReceivable has a positive value, albeit in the PDF filing it is displayed as negative, and there is a <link:calculationArc order="6" "weight="-1.0" Not knowing the sign up a value is a pretty big challenge to overcome ;)
XBRL also has a "balance_type" (credit, debit) that may also affect a value's sign
XBRL has valuable hierarchical information that shows parent/ child relationships. It is a pity that this information is completely lost in to_json() process

Regards, Johannes

I know this was raised a bug before, and was closed as an "accounting detail", but I am still unsure how to proceed. I wrote some code to deduce sign from calculation rules with following output (again for AAPL 2021-09-25):

Prefix "-" means that calculation weight was "-1.0".

In some cases even negative weight for an already negative value:

    -OtherComprehensiveIncomeLossDerivativeInstrumentGainLossReclassificationAfterTax: -1003000000.0

1001002 - Statement - CONSOLIDATED STATEMENTS OF OPERATIONS

GrossProfit: 152836000000.0
    RevenueFromContractWithCustomerExcludingAssessedTax: 365817000000.0
    -CostOfGoodsAndServicesSold: 212981000000.0
NetIncomeLoss: 94680000000.0
    IncomeLossFromContinuingOperationsBeforeIncomeTaxesExtraordinaryItemsNoncontrollingInterest: 109207000000.0
    -IncomeTaxExpenseBenefit: 14527000000.0
OperatingExpenses: 43887000000.0
    ResearchAndDevelopmentExpense: 21914000000.0
    SellingGeneralAndAdministrativeExpense: 21973000000.0
IncomeLossFromContinuingOperationsBeforeIncomeTaxesExtraordinaryItemsNoncontrollingInterest: 109207000000.0
    OperatingIncomeLoss: 108949000000.0
    NonoperatingIncomeExpense: 258000000.0
OperatingIncomeLoss: 108949000000.0
    GrossProfit: 152836000000.0
    -OperatingExpenses: 43887000000.0

1002003 - Statement - CONSOLIDATED STATEMENTS OF COMPREHENSIVE INCOME

OtherComprehensiveIncomeLossAvailableForSaleSecuritiesAdjustmentNetOfTax: -967000000.0
    OtherComprehensiveIncomeUnrealizedHoldingGainLossOnSecuritiesArisingDuringPeriodNetOfTax: -694000000.0
    -OtherComprehensiveIncomeLossReclassificationAdjustmentFromAOCIForSaleOfSecuritiesNetOfTax: 273000000.0
ComprehensiveIncomeNetOfTax: 95249000000.0
    NetIncomeLoss: 94680000000.0
    OtherComprehensiveIncomeLossNetOfTaxPortionAttributableToParent: 569000000.0
OtherComprehensiveIncomeLossDerivativeInstrumentGainLossafterReclassificationandTax: 1035000000.0
    OtherComprehensiveIncomeLossDerivativeInstrumentGainLossbeforeReclassificationafterTax: 32000000.0
    -OtherComprehensiveIncomeLossDerivativeInstrumentGainLossReclassificationAfterTax: -1003000000.0
OtherComprehensiveIncomeLossNetOfTaxPortionAttributableToParent: 569000000.0
    OtherComprehensiveIncomeLossForeignCurrencyTransactionAndTranslationAdjustmentNetOfTax: 501000000.0
    OtherComprehensiveIncomeLossDerivativeInstrumentGainLossafterReclassificationandTax: 1035000000.0
    OtherComprehensiveIncomeLossAvailableForSaleSecuritiesAdjustmentNetOfTax: -967000000.0

1003004 - Statement - CONSOLIDATED BALANCE SHEETS

LiabilitiesCurrent: 125481000000.0
    AccountsPayableCurrent: 54763000000.0
    OtherLiabilitiesCurrent: 47493000000.0
    ContractWithCustomerLiabilityCurrent: 7612000000.0
    CommercialPaper: 6000000000.0
    LongTermDebtCurrent: 9613000000.0
AssetsNoncurrent: 216166000000.0
    MarketableSecuritiesNoncurrent: 127877000000.0
    PropertyPlantAndEquipmentNet: 39440000000.0
    OtherAssetsNoncurrent: 48849000000.0
LiabilitiesAndStockholdersEquity: 351002000000.0
    Liabilities: 287912000000.0
    CommitmentsAndContingencies: None
    StockholdersEquity: 63090000000.0
Liabilities: 287912000000.0
    LiabilitiesCurrent: 125481000000.0
    LiabilitiesNoncurrent: 162431000000.0
AssetsCurrent: 134836000000.0
    CashAndCashEquivalentsAtCarryingValue: 34940000000.0
    MarketableSecuritiesCurrent: 27699000000.0
    AccountsReceivableNetCurrent: 26278000000.0
    InventoryNet: 6580000000.0
    NontradeReceivablesCurrent: 25228000000.0
    OtherAssetsCurrent: 14111000000.0
Assets: 351002000000.0
    AssetsCurrent: 134836000000.0
    AssetsNoncurrent: 216166000000.0
LiabilitiesNoncurrent: 162431000000.0
    LongTermDebtNoncurrent: 109106000000.0
    OtherLiabilitiesNoncurrent: 53325000000.0
StockholdersEquity: 63090000000.0
    CommonStocksIncludingAdditionalPaidInCapital: 57365000000.0
    RetainedEarningsAccumulatedDeficit: 5562000000.0
    AccumulatedOtherComprehensiveIncomeLossNetOfTax: 163000000.0

1006007 - Statement - CONSOLIDATED STATEMENTS OF CASH FLOWS

NetCashProvidedByUsedInOperatingActivities: 104038000000.0
    NetIncomeLoss: 94680000000.0
    DepreciationDepletionAndAmortization: 11284000000.0
    ShareBasedCompensation: 7906000000.0
    DeferredIncomeTaxExpenseBenefit: -4774000000.0
    -OtherNoncashIncomeExpense: 147000000.0
    -IncreaseDecreaseInAccountsReceivable: 10125000000.0
    -IncreaseDecreaseInInventories: 2642000000.0
    -IncreaseDecreaseInOtherReceivables: 3903000000.0
    -IncreaseDecreaseInOtherOperatingAssets: 8042000000.0
    IncreaseDecreaseInAccountsPayable: 12326000000.0
    IncreaseDecreaseInContractWithCustomerLiability: 1676000000.0
    IncreaseDecreaseInOtherOperatingLiabilities: 5799000000.0
CashCashEquivalentsRestrictedCashAndRestrictedCashEquivalentsPeriodIncreaseDecreaseIncludingExchangeRateEffect: -3860000000.0
    NetCashProvidedByUsedInOperatingActivities: 104038000000.0
    NetCashProvidedByUsedInInvestingActivities: -14545000000.0
    NetCashProvidedByUsedInFinancingActivities: -93353000000.0
NetCashProvidedByUsedInFinancingActivities: -93353000000.0
    ProceedsFromIssuanceOfCommonStock: 1105000000.0
    -PaymentsRelatedToTaxWithholdingForShareBasedCompensation: 6556000000.0
    -PaymentsOfDividends: 14467000000.0
    -PaymentsForRepurchaseOfCommonStock: 85971000000.0
    ProceedsFromIssuanceOfLongTermDebt: 20393000000.0
    -RepaymentsOfLongTermDebt: 8750000000.0
    ProceedsFromRepaymentsOfCommercialPaper: 1022000000.0
    ProceedsFromPaymentsForOtherFinancingActivities: -129000000.0
NetCashProvidedByUsedInInvestingActivities: -14545000000.0
    -PaymentsToAcquireAvailableForSaleSecuritiesDebt: 109558000000.0
    ProceedsFromMaturitiesPrepaymentsAndCallsOfAvailableForSaleSecurities: 59023000000.0
    ProceedsFromSaleOfAvailableForSaleSecuritiesDebt: 47460000000.0
    -PaymentsToAcquirePropertyPlantAndEquipment: 11085000000.0
    -PaymentsToAcquireBusinessesNetOfCashAcquired: 33000000.0
    -PaymentsToAcquireOtherInvestments: 131000000.0
    ProceedsFromSaleAndMaturityOfOtherInvestments: 387000000.0
    -PaymentsForProceedsFromOtherInvestingActivities: 608000000.0

2405402 - Disclosure - Summary of Significant Accounting Policies - Computation of Basic and Diluted Earnings Per Share (Details)

WeightedAverageNumberOfDilutedSharesOutstanding: 16864919000.0
    WeightedAverageNumberOfSharesOutstandingBasic: 16701272000.0
    WeightedAverageNumberDilutedSharesOutstandingAdjustment: 163647000.0

2413406 - Disclosure - Financial Instruments - Cash, Cash Equivalents and Marketable Securities (Details)

CashCashEquivalentsAndMarketableSecurities: 190516000000.0
    CashAndCashEquivalentsAtCarryingValue: 34940000000.0
    MarketableSecuritiesCurrent: 27699000000.0
    MarketableSecuritiesNoncurrent: 127877000000.0
CashEquivalentsAndMarketableSecuritiesAccumulatedGrossUnrealizedGainBeforeTax: 1753000000.0
CashCashEquivalentsAndMarketableSecuritiesCost: 189961000000.0
CashEquivalentsAndMarketableSecuritiesAccumulatedGrossUnrealizedLossBeforeTax: 1198000000.0

2413406 - Disclosure - Financial Instruments - Cash, Cash Equivalents and Marketable Securities (Details)

CashCashEquivalentsAndMarketableSecurities: 190516000000.0
CashCashEquivalentsAndMarketableSecuritiesCost: 189961000000.0
    -CashEquivalentsAndMarketableSecuritiesAccumulatedGrossUnrealizedGainBeforeTax: 1753000000.0
    CashEquivalentsAndMarketableSecuritiesAccumulatedGrossUnrealizedLossBeforeTax: 1198000000.0
    CashCashEquivalentsAndMarketableSecurities: 190516000000.0

2414407 - Disclosure - Financial Instruments - Non-Current Marketable Debt Securities by Contractual Maturity (Details)

AvailableForSaleSecuritiesDebtMaturitiesSingleMaturityDate: 127877000000.0
    AvailableForSaleSecuritiesDebtMaturitiesRollingYearTwoThroughFiveFairValue: 83755000000.0
    AvailableForSaleSecuritiesDebtMaturitiesRollingYearSixThroughTenFairValue: 23915000000.0
    AvailableForSaleSecuritiesDebtMaturitiesRollingAfterYearTenFairValue: 20207000000.0

2420411 - Disclosure - Consolidated Financial Statement Details - Property, Plant and Equipment, Net (Details)

PropertyPlantAndEquipmentNet: 39440000000.0
    PropertyPlantAndEquipmentGross: 109723000000.0
    -AccumulatedDepreciationDepletionAndAmortizationPropertyPlantAndEquipment: 70283000000.0

2421412 - Disclosure - Consolidated Financial Statement Details - Other Non-Current Liabilities (Details)

OtherLiabilitiesNoncurrent: 53325000000.0
    AccruedIncomeTaxesNoncurrent: 24689000000.0
    OtherAccruedLiabilitiesNoncurrent: 28636000000.0

2422413 - Disclosure - Consolidated Financial Statement Details - Other Income/(Expense), Net (Details)

NonoperatingIncomeExpense: 258000000.0
    InvestmentIncomeInterestAndDividend: 2843000000.0
    -InterestExpense: 2645000000.0
    OtherNonoperatingIncomeExpense: 60000000.0

2425414 - Disclosure - Income Taxes - Provision for Income Taxes (Details)

IncomeTaxExpenseBenefit: 14527000000.0
    FederalIncomeTaxExpenseBenefitContinuingOperations: 1081000000.0
    StateAndLocalIncomeTaxExpenseBenefitContinuingOperations: 1282000000.0
    ForeignIncomeTaxExpenseBenefitContinuingOperations: 12164000000.0
FederalIncomeTaxExpenseBenefitContinuingOperations: 1081000000.0
    CurrentFederalTaxExpenseBenefit: 8257000000.0
    DeferredFederalIncomeTaxExpenseBenefit: -7176000000.0
StateAndLocalIncomeTaxExpenseBenefitContinuingOperations: 1282000000.0
    CurrentStateAndLocalTaxExpenseBenefit: 1620000000.0
    DeferredStateAndLocalIncomeTaxExpenseBenefit: -338000000.0
ForeignIncomeTaxExpenseBenefitContinuingOperations: 12164000000.0
    CurrentForeignTaxExpenseBenefit: 9424000000.0
    DeferredForeignIncomeTaxExpenseBenefit: 2740000000.0

2427416 - Disclosure - Income Taxes - Reconciliation of the Provision for Income Taxes (Details)

IncomeTaxExpenseBenefit: 14527000000.0
    IncomeTaxReconciliationIncomeTaxExpenseBenefitAtFederalStatutoryIncomeTaxRate: 22933000000.0
    IncomeTaxReconciliationStateAndLocalIncomeTaxes: 1151000000.0
    EffectiveIncomeTaxRateReconciliationTaxCutsAndJobsActOf2017Amount: 0.0
    IncomeTaxReconciliationForeignIncomeTaxRateDifferential: -4715000000.0
    -EffectiveIncomeTaxRateReconciliationForeignDerivedIntangibleIncomeDeductionAmount: 1372000000.0
    -IncomeTaxReconciliationTaxCreditsResearch: 1033000000.0
    EffectiveIncomeTaxRateReconciliationShareBasedCompensationExcessTaxBenefitAmount: -2137000000.0
    IncomeTaxReconciliationOtherAdjustments: -300000000.0

2428417 - Disclosure - Income Taxes - Significant Components of Deferred Tax Assets and Liabilities (Details)

DeferredTaxAssetsNet: 20273000000.0
    DeferredTaxAssetsGross: 25176000000.0
    -DeferredTaxAssetsValuationAllowance: 4903000000.0
DeferredTaxAssetsGross: 25176000000.0
    DeferredTaxAssetsGoodwillAndIntangibleAssets: 5575000000.0
    DeferredTaxAssetsTaxDeferredExpenseReservesAndAccruals: 5895000000.0
    DeferredTaxAssetsLeaseLiabilities: 2406000000.0
    DeferredTaxAssetsDeferredIncome: 5399000000.0
    DeferredTaxAssetsTaxCreditCarryforwards: 4262000000.0
    DeferredTaxAssetsOther: 1639000000.0
DeferredIncomeTaxLiabilities: 7200000000.0
    DeferredTaxLiabilitiesMinimumTaxonForeignEarnings: 4318000000.0
    DeferredTaxLiabilitiesLeasingArrangements: 2167000000.0
    DeferredTaxLiabilitiesOtherComprehensiveIncome: 203000000.0
    DeferredTaxLiabilitiesOther: 512000000.0
DeferredTaxAssetsLiabilitiesNet: 13073000000.0
    DeferredTaxAssetsNet: 20273000000.0
    -DeferredIncomeTaxLiabilities: 7200000000.0

2433420 - Disclosure - Leases - ROU Assets and Lease Liabilities (Details)

OperatingandFinanceLeaseLiability: 11803000000.0
    OperatingLeaseLiabilityCurrent: 1449000000.0
    OperatingLeaseLiabilityNoncurrent: 9506000000.0
    FinanceLeaseLiabilityCurrent: 79000000.0
    FinanceLeaseLiabilityNoncurrent: 769000000.0
OperatingandFinanceLeaseRightofUseAsset: 10948000000.0
    OperatingLeaseRightOfUseAsset: 10087000000.0
    FinanceLeaseRightOfUseAsset: 861000000.0

2434421 - Disclosure - Leases - Lease Liability Maturities (Details)

LesseeOperatingandFinanceLeaseLiabilityPaymentsDue: 13453000000.0
    LesseeOperatingandFinanceLeaseLiabilityUndiscountedExcessAmount: 1650000000.0
    OperatingandFinanceLeaseLiability: 11803000000.0
LesseeOperatingAndFinanceLeaseLiabilityToBePaidYearTwo: 1683000000.0
    LesseeOperatingLeaseLiabilityPaymentsDueYearTwo: 1560000000.0
    FinanceLeaseLiabilityPaymentsDueYearTwo: 123000000.0
LesseeOperatingAndFinanceLeaseLiabilityToBePaidYearOne: 1733000000.0
    LesseeOperatingLeaseLiabilityPaymentsDueNextTwelveMonths: 1629000000.0
    FinanceLeaseLiabilityPaymentsDueNextTwelveMonths: 104000000.0
LesseeOperatingLeaseLiabilityPaymentsDue: 12187000000.0
    LesseeOperatingLeaseLiabilityUndiscountedExcessAmount: 1232000000.0
    OperatingLeaseLiability: 10955000000.0
FinanceLeaseLiabilityPaymentsDue: 1266000000.0
    FinanceLeaseLiabilityUndiscountedExcessAmount: 418000000.0
    FinanceLeaseLiability: 848000000.0
LesseeOperatingandFinanceLeaseLiabilityUndiscountedExcessAmount: 1650000000.0
    LesseeOperatingLeaseLiabilityUndiscountedExcessAmount: 1232000000.0
    FinanceLeaseLiabilityUndiscountedExcessAmount: 418000000.0
LesseeOperatingAndFinanceLeaseLiabilityToBePaidYearThree: 1598000000.0
    LesseeOperatingLeaseLiabilityPaymentsDueYearThree: 1499000000.0
    FinanceLeaseLiabilityPaymentsDueYearThree: 99000000.0
LesseeOperatingAndFinanceLeaseLiabilityToBePaidAfterYearFive: 6055000000.0
    LesseeOperatingLeaseLiabilityPaymentsDueAfterYearFive: 5187000000.0
    FinanceLeaseLiabilityPaymentsDueAfterYearFive: 868000000.0
OperatingandFinanceLeaseLiability: 11803000000.0
    OperatingLeaseLiability: 10955000000.0
    FinanceLeaseLiability: 848000000.0
LesseeOperatingAndFinanceLeaseLiabilityToBePaidYearFive: 1087000000.0
    LesseeOperatingLeaseLiabilityPaymentsDueYearFive: 1061000000.0
    FinanceLeaseLiabilityPaymentsDueYearFive: 26000000.0
LesseeOperatingAndFinanceLeaseLiabilityToBePaidYearFour: 1297000000.0
    LesseeOperatingLeaseLiabilityPaymentsDueYearFour: 1251000000.0
    FinanceLeaseLiabilityPaymentsDueYearFour: 46000000.0

2434421 - Disclosure - Leases - Lease Liability Maturities (Details)

FinanceLeaseLiabilityPaymentsDue: 1266000000.0
    FinanceLeaseLiabilityPaymentsDueNextTwelveMonths: 104000000.0
    FinanceLeaseLiabilityPaymentsDueYearTwo: 123000000.0
    FinanceLeaseLiabilityPaymentsDueYearThree: 99000000.0
    FinanceLeaseLiabilityPaymentsDueYearFour: 46000000.0
    FinanceLeaseLiabilityPaymentsDueYearFive: 26000000.0
    FinanceLeaseLiabilityPaymentsDueAfterYearFive: 868000000.0
LesseeOperatingLeaseLiabilityPaymentsDue: 12187000000.0
    LesseeOperatingLeaseLiabilityPaymentsDueNextTwelveMonths: 1629000000.0
    LesseeOperatingLeaseLiabilityPaymentsDueYearTwo: 1560000000.0
    LesseeOperatingLeaseLiabilityPaymentsDueYearThree: 1499000000.0
    LesseeOperatingLeaseLiabilityPaymentsDueYearFour: 1251000000.0
    LesseeOperatingLeaseLiabilityPaymentsDueYearFive: 1061000000.0
    LesseeOperatingLeaseLiabilityPaymentsDueAfterYearFive: 5187000000.0
LesseeOperatingandFinanceLeaseLiabilityPaymentsDue: 13453000000.0
    LesseeOperatingLeaseLiabilityPaymentsDue: 12187000000.0
    FinanceLeaseLiabilityPaymentsDue: 1266000000.0

2434421 - Disclosure - Leases - Lease Liability Maturities (Details)

LesseeOperatingandFinanceLeaseLiabilityPaymentsDue: 13453000000.0
    LesseeOperatingAndFinanceLeaseLiabilityToBePaidYearOne: 1733000000.0
    LesseeOperatingAndFinanceLeaseLiabilityToBePaidYearTwo: 1683000000.0
    LesseeOperatingAndFinanceLeaseLiabilityToBePaidYearThree: 1598000000.0
    LesseeOperatingAndFinanceLeaseLiabilityToBePaidYearFour: 1297000000.0
    LesseeOperatingAndFinanceLeaseLiabilityToBePaidYearFive: 1087000000.0
    LesseeOperatingAndFinanceLeaseLiabilityToBePaidAfterYearFive: 6055000000.0

2438423 - Disclosure - Debt - Summary of Cash Flows Associated with Commercial Paper (Details)

ProceedsFromRepaymentsOfCommercialPaper: 1022000000.0
    ProceedsFromRepaymentsOfShortTermDebtMaturingInThreeMonthsOrLess: -357000000.0
    ProceedsFromRepaymentsOfShortTermDebtMaturingInMoreThanThreeMonths: 1379000000.0
ProceedsFromRepaymentsOfShortTermDebtMaturingInMoreThanThreeMonths: 1379000000.0
    ProceedsFromShortTermDebtMaturingInMoreThanThreeMonths: 7946000000.0
    -RepaymentsOfShortTermDebtMaturingInMoreThanThreeMonths: 6567000000.0

2440425 - Disclosure - Debt - Future Principal Payments for Term Debt (Details)

DebtInstrumentCarryingAmount: 118063000000.0
    LongTermDebtMaturitiesRepaymentsOfPrincipalInNextTwelveMonths: 9583000000.0
    LongTermDebtMaturitiesRepaymentsOfPrincipalInYearTwo: 11391000000.0
    LongTermDebtMaturitiesRepaymentsOfPrincipalInYearThree: 10202000000.0
    LongTermDebtMaturitiesRepaymentsOfPrincipalInYearFour: 10914000000.0
    LongTermDebtMaturitiesRepaymentsOfPrincipalInYearFive: 11408000000.0
    LongTermDebtMaturitiesRepaymentsOfPrincipalAfterYearFive: 64565000000.0

2453432 - Disclosure - Commitments and Contingencies - Future Payments Under Unconditional Purchase Obligations (Details)

UnrecordedUnconditionalPurchaseObligationBalanceSheetAmount: 8184000000.0
    UnrecordedUnconditionalPurchaseObligationBalanceOnFirstAnniversary: 4551000000.0
    UnrecordedUnconditionalPurchaseObligationBalanceOnSecondAnniversary: 2165000000.0
    UnrecordedUnconditionalPurchaseObligationBalanceOnThirdAnniversary: 984000000.0
    UnrecordedUnconditionalPurchaseObligationBalanceOnFourthAnniversary: 405000000.0
    UnrecordedUnconditionalPurchaseObligationBalanceOnFifthAnniversary: 51000000.0
    UnrecordedUnconditionalPurchaseObligationDueAfterFiveYears: 28000000.0

Apologies for the spam, I am also only figuring this out as I go along.

(source: https://www.wisesheets.io/assets/PreparersGuide.pdf)

@codinguncut As the pdf says in the previous comment, the weight is assigned to a particular calculation relationship, not the value. Depending on what is being calculated, a particular value could be added or subtracted (which is why it's modeled this way).

One way you can tackle this is to parse the CALCULATION Linkbase as well as the filing. Once you have the calculation linkbase you can iterate over every concept in the calc linkbase, for each concept, fill in the value you found in the facts lists and you're set.

I've been working through this exact exercise over the past couple weeks, so lemme know if we can help each other.

@manusimidt please correct me if you've developed a different way to do this.

Thank you both for your suggestions!

Yes, if i understand correctly, the calculation linkbase does not indicate which sign the fact has, but how it relates to other facts. But I understand the confusion here, if you look for example at this fact, even though it appears to be negative on the actual cash flow statement, it has a positive sign in XBRL. This is probably because it's concept has creditor character. You can also directly access this from the fact object instance (fact.concept.balance is either 'credit' or 'debit').

Furthermore the calculation linkbase shows you that the one fact is substracted from the other:

I know that its currently very tedious to work with the linkbases in py-xbrl because it only extract the information from the linkbases and basically just stores it in a hierarchical structure and doesn't use it further. In the beginning I always wanted to compile the linkbases down so that every bit of information can be easily extracted from the XbrlInstance object. Unfortunately, an accurate implemation turned out to be much more difficult than I originally thought. Mainly due to the fact that taxonomies can have complicated relationships and can override, forbid relations of other taxonomies or can add new relations/ressources.

A feature that also was requested by many people was to compile the presentation linkbase. So that i.e.: you call a function of the XbrlInstance object instance and get back a hierachical structure containing all Facts and their labels as they appear on the printed version of the XBRL document. A similar function could be made for the calculation linkbase.

Tomorrow I will look further into the exact submission (AAPL 2021-09) @codinguncut pointed out and think about solutions. Unfortunately, I currently only find time to implement new features in the libary during the semester break because I study full time and work part time on the side, but I'll still take a closer look at the issue tomorrow :)

@manusimidt I've been grappling with this issue as well. Specifically where the filer overrides the base calculations which happens in all the filings I've seen so far (a few dozen).

I'm only really interested in the us-gaap taxonomies, but I believe the following implementation should follow for all.

Currently my implementation grabs what I call the referenced base taxonomy. For example, FilingA references the us-gaap 2022 taxonomy. For which I parse every calculation linkbase in the base taxonomy. Then I take FilingA's calculation linkbase and replace every concept defined in the base with FilingA's definition.

So imagine the us-gaap has us-gaap_NetIncomeLoss = us-gaap_ProfitLoss + us-gaap_SomeOtherIncomeSource + ... And FilingA has definition for us-gaap_NetIncomeLoss = us-gaap_OperatingIncomeLoss + us-gaap_InterestExpense + ...

I replace the definition in the us-gaap with FilingA's definition.

In the next week or so I'll be running this across a large set of filings to confirm my implementation is roughly correct, but lemme know if you see anything glaringly wrong about this or if you need any further clarification.

hi @ajmed @manusimidt, thank you so much for your detailed comments. Once I understand the correct behavior, I would also be happy to implement a solution in this library, or to collaborate on doing so.

I don't quite understand the relationship between the base schema and "overrides". I would have thought that "FilingA definition" would be a subset of the "us_gaap definition", otherwise the value of the whole XBLR thing would be a bit dubious.

https://xbrl.us/forums/topic/how-to-find-a-complete-list-of-similar-concept/

" even after spending a tremendous amount of time trying to standardize the tags, I have failed to come up with a workable solution."
"After reading this thread, am I correct in concluding there is no standard IFRS/GAAP/ect standard for pulling data across periods in a company and across companies? "
"I gave up scraping the 10-K/Q for just the reason mentioned. The XBRL tag names change over time. So I went back to Yahoo! and Alpha Vantage."

Generally, regulators decide between one of two different reporting models. The Open financial reporting (like SEC uses it) and the closed financial reporting (like the company house in the UK uses it).

In a closed reporting model the company must have the common taxonomy (IFRS for UK) as base taxonomy and is not allowed to add new concepts, relationships and resources.

In an open reporting model the company is allowed to create its own extension taxonomy for every submission. The most common thing companies do in their extension taxonomy is to add new concepts, resources and relationships.

For example: Apple might add a new concept "aapl_iPadMember", add the label (resource) "Total Iphones sold" and add the relationship (calculation) "us-gaap_TotalProductMember = aapl_iPhoneMember + aapl_iPadMember"

To be honest I have never seen a submission that overrides important concepts of the base taxonomy. I mostly worked with SEC submissions and the us-gaap taxonomy which they import usually also has no presentation or calculation linkbase, which only leaves the calculation/presentation linkbases that the company provides. But according to the Specification, it is possible and as far as I know, the ESEF Taxonomy does it. (ESEF is the new European taxonomy that inherits from IFRS.)

But yes, @codinguncut you definitely have a point. I also run into this issue some times. But I can tell you that from my experience most of the companies use common us-gaap tax on their three big financial statements (Cash-flow, Balance Sheet, Income Statement).

So to summarize: Overriding relations in extension taxonomies is complex and theoretically a thing but not really relevant for SEC submissions (according to my experience). What is way more important is to capture the newly added concepts and relationships.

I can also really recommend the book "XBRL for interactive Data" from Debreceny. It also has a chapter on this topic and it is explained quite well. (The following figure is from this book)

@ajmed yes, this sounds pretty interesting! I looking forward to hear from you about your results!

I have also often thought about how I can parse this in the easiest way. Until now I thought to make an artificial "final taxonomy", go recursively through all imports, start at the deepest taxonomy and gradually add all resources/relations to the "final taxonomy". If a relation is overwritten in a "higher" taxonomy, this would then also have to be overwritten in the "final taxonomy".

The goal of the whole thing is to end up with just one taxonomy that is easy to work with and not (as it is currently) an import array in which further taxonomies are stored which in turn has also an import array that can contain further taxonomies (and so on).

@manusimidt That's exactly as I have it currently. Still a work in progress and only focused on the us-gaap, but lemme provide an example filing: 0001090872-22-000012. Specifically you can open the calc linkbase and ctrl-f for us-gaap_NetIncomeLoss, below I have the relevant concept highlighted:

{
    "us-gaap_NetIncomeLoss": {
    "children": [
      {
        "concept_id": "us-gaap_IncomeLossFromContinuingOperationsBeforeIncomeTaxesExtraordinaryItemsNoncontrollingInterest",
        "weight": 1.0
      },
      {
        "concept_id": "us-gaap_IncomeTaxExpenseBenefit",
        "weight": -1.0
      }
    ]
  }
}

As I understand it, the above is a complete override of the base us-gaap_NetIncomeLoss calculation. Specifically: us-gaap-stm-soi-cal-2022.xml

"us-gaap_NetIncomeLoss": {
      "children": [
        {
          "us-gaap_ProfitLoss": 10
        },
        {
          "us-gaap_NetIncomeLossAttributableToNoncontrollingInterest": 20
        }
      ],
}

Sorry for the inconsistent formats. The former is redacted output from this library and the latter is how I've been parsing the calc linkbases so I get constant lookup by concept id.

In any case, now, when I want to report to a business user the us-gaap_NetIncomeLoss of a filing. I use the calculation from the filer's calc linkbase instead of the us-gaap linkbase.

To be explicit, I'm using the calculation: us-gaap_NetIncomeLoss = us-gaap_IncomeLossFromContinuingOperationsBeforeIncomeTaxesExtraordinaryItemsNoncontrollingInterest + (- us-gaap_IncomeTaxExpenseBenefit)

(Notice the minus sign on the second term from the weight.)

@manusimidt Regarding Closed reporting, it appears that at least in Denmark it is possible to add concepts to IFRS-Full (parken:ResultFromOperatingActivitiesBeforeAmortizationTransferActivitiesAndSpecialItems):

Also in the UK (procookgroupplc:BonusIssue):

I am really confused by the other tabular data formats provided by Edgar (xbrl-json, tsv, etc.)

https://data.sec.gov/api/xbrl/companyfacts/CIK0000320193.json
https://www.sec.gov/files/dera/data/financial-statement-data-sets/2022q3.zip (at least the DERA data has some dubious calculation and presentation information)

If filers can arbitrarily assign "+" or "-" signs to their facts, or completely change calculation logic, then all of the above would be worthless because the sign to be used would not be clear from the json or tsv data. Not even to mention that at least the json data seems to completely throw away the schema versioning.

Nice tool, also shows how much companies extend the base XBRL schema: https://edgardashboard.xbrlcloud.com/edgar-dashboard/

@ajmed Yes, in theory, it exactly works that way. However, what puzzles me is that i can't see where the calculation linkbase (us-gaap-stm-soi-cal-2022.xml) is imported by the filing (0001090872-22-000012). The schema file of the filing imports the following endpoint of the us-gaap taxonomy: us-gaap-2021-01-31.xsd. However this schema file does not import any external linkbases.

Disclaimer: I am also not really familiar with the taxonomy architecture. But I try my best:

Big taxonomies like the us-gaap have different entry points. If you i.e.: look at the taxonomy directory of the us-gaap 2021 you will see five different folders: As far as I understand it, correlate these folders with different parts of the architecture. If you import the the endpoint (entire/us-gaap-entryPoint-all-2021-01-31.xsd) in your schema you will have all concepts and linkbases applied to your submission. SEC Edgar submissions however always import the main schema file in the /elts folder (i.e: /elts/us-gaap-2021-01-31.xsd. This file does not import any calculation linkbases.

I don't know exactly why but this is probably prescribed by the regulator.

For some reason, i couldn't find a single file/website describing the architecture of the us-gaap taxonomy. I could only find one for the european ESEF taxonomy that was introduced last year:

I would understand it this way: if you import the full taxonomy (èsef_all.xsd), all linkbases (esef_all-cal.xml, esel_all-def.xml...) apply to you. If you import just the core taxonomy (esef_cor.xsd) the linkbases would not apply to your instance file because they are not directly imported.

@codinguncut "On a case-by-case basis" 😂🤦🏼‍♂️ defeats the idea of XBRL a bit... Thank you for the EDGAR Dashboard link, looks very interesting.

manusimidt / py-xbrl

Unclear +/- sign of some facts #91