portfolio-performance / portfolio

Track and evaluate the performance of your investment portfolio across stocks, cryptocurrencies, and other assets.
http://www.portfolio-performance.info
Eclipse Public License 1.0
2.89k stars 599 forks source link

IBFlex-Query with futures & commodities #3742

Open rektosaure opened 8 months ago

rektosaure commented 8 months ago

Describe the bug It appears that futures and commodities trades are not being detected by the importer.

In IBFlexStatementExtractor.java, and more specifically in IBFlexStatementExtractorResult, we should find the constants ASSETKEY_STOCK = "FUT" and ASSETKEY_STOCK = "CMDTY"

Below is an example of an FQ extract: Future:

<Trade accountId="xx" acctAlias="xx" model="" currency="EUR" fxRateToBase="1" assetCategory="FUT" subCategory="" symbol="FDXS 20240315 M" description="DAX 15MAR24" conid="638120377" securityID="" securityIDType="" cusip="" isin="" figi="BBG01H30WMN8" listingExchange="EUREX" underlyingConid="825711" underlyingSymbol="DAX" underlyingSecurityID="" underlyingListingExchange="" issuer="" multiplier="1" tradeID="xx" strike="" relatedTradeID="" expiry="20240315" reportDate="20231214" putCall="" dateTime="20231214;121646" principalAdjustFactor="" tradeDate="20231214" settleDateTarget="20231215" transactionType="ExchTrade" exchange="EUREX" quantity="-1" tradePrice="16969" tradeMoney="-16969" proceeds="16969" taxes="0" ibCommission="-0.4" ibCommissionCurrency="EUR" netCash="8.6" closePrice="16960" openCloseIndicator="O" notes="" cost="-16968.6" fifoPnlRealized="0" mtmPnl="9" origTradePrice="0" origTradeDate="" origTradeID="" origOrderID="0" origTransactionID="0" buySell="SELL" clearingFirmID="" ibOrderID="xx" transactionID="xx" ibExecID="xx" relatedTransactionID="" brokerageOrderID="" orderReference="" volatilityOrderLink="" exchOrderId="N/A" extExecID="N/A" orderTime="20231214;121646" openDateTime="" holdingPeriodDateTime="" whenRealized="" whenReopened="" levelOfDetail="EXECUTION" changeInPrice="0" changeInQuantity="0" orderType="" traderID="" isAPIOrder="N" accruedInt="0" serialNumber="" deliveryType="" commodityType="" fineness="0.0" weight="0.0" issuerCountryCode="" />

Commodity:

<Trade accountId="xx" acctAlias="xx" model="" currency="USD" fxRateToBase="0.92283" assetCategory="CMDTY" subCategory="" symbol="XAUUSD" description="London Gold" conid="69067924" securityID="" securityIDType="" cusip="" isin="" figi="" listingExchange="" underlyingConid="" underlyingSymbol="" underlyingSecurityID="" underlyingListingExchange="" issuer="" multiplier="1" tradeID="xx" strike="" relatedTradeID="" expiry="" reportDate="20231204" putCall="" dateTime="20231203;182345" principalAdjustFactor="" tradeDate="20231204" settleDateTarget="20231206" transactionType="ExchTrade" exchange="UBSFX" quantity="-2" tradePrice="2115.3" tradeMoney="-4230.6" proceeds="4230.6" taxes="0" ibCommission="-1" ibCommissionCurrency="USD" netCash="4229.6" closePrice="2029.41" openCloseIndicator="C" notes="" cost="-4017.5" fifoPnlRealized="212.1" mtmPnl="171.78" origTradePrice="0" origTradeDate="" origTradeID="" origOrderID="0" origTransactionID="0" buySell="SELL" clearingFirmID="" ibOrderID="546728647" transactionID="2316076751" ibExecID="0002f339.656d0e04.01.01" relatedTransactionID="" brokerageOrderID="" orderReference="" volatilityOrderLink="" exchOrderId="N/A" extExecID="N/A" orderTime="20231203;182345" openDateTime="" holdingPeriodDateTime="" whenRealized="" whenReopened="" levelOfDetail="EXECUTION" changeInPrice="0" changeInQuantity="0" orderType="" traderID="" isAPIOrder="N" accruedInt="0" serialNumber="" deliveryType="" commodityType="" fineness="0.0" weight="0.0" issuerCountryCode="" />

To Reproduce Steps to reproduce the behavior:

  1. Buy or sell future contract and commodity
  2. Run Flex Query extract
  3. Import xml file to PP
  4. Future and commodity trades are missing

Expected behavior Future and commodity trades should be imported

Desktop (please complete the following information):

Nirus2000 commented 8 months ago

Hello @arnishow Thank you for reporting the issue. However, we need a complete *.xml file with all content. So a complete one, with please a few transactions and all existing elements. (FlexQueryResponse, FlexStatements, CashReport, Trades, CashTransactions, InterestAccruals etc.) The personal data must of course be anonymized.

Furthermore, please explain what is to be posted. So example <--> result.

Regards Alex

rektosaure commented 8 months ago

Hi Alex, Here is a sample extract PP.txt

Nirus2000 commented 8 months ago

Hello @arnishow Can you explain what that is individually? I don't have an IB and can't do anything with the abbreviations.

What should be posted? (Purchase, sale, fees... I don't know)

Please in an example.

Regards Alex

rektosaure commented 8 months ago

I'm not sure to understand exactly what you need.

After few tests, it looks like commodities (CMDTY) works exactly as stocks (STK) and should be easy to implement by adding ASSETKEY_STOCK = "CMDTY" in IBFlexStatementExtractorResult.

Nirus2000 commented 8 months ago

Hello @arnishow I do not have an IB-Flex and I have no experience with FUT or CMDTY.

I need the following information from you:

1. FUT <Trade accountId="xxx" acctAlias="xxx" model="" currency="EUR" fxRateToBase="1" assetCategory="FUT" subCategory="" symbol="FDXS 20240315 M" description="DAX 15MAR24" conid="638120377" securityID="" securityIDType="" cusip="" isin="" figi="BBG01H30WMN8" listingExchange="EUREX" underlyingConid="825711" underlyingSymbol="DAX" underlyingSecurityID="" underlyingListingExchange="" issuer="" issuerCountryCode="" tradeID="651998678" multiplier="1" relatedTradeID="" strike="" reportDate="20231214" expiry="20240315" dateTime="20231214;092301" putCall="" tradeDate="20231214" principalAdjustFactor="" settleDateTarget="20231215" transactionType="ExchTrade" exchange="EUREX" quantity="-1" tradePrice="17032" tradeMoney="-17032" proceeds="17032" taxes="0" ibCommission="-0.4" ibCommissionCurrency="EUR" netCash="71.6" closePrice="16960" openCloseIndicator="O" notes="" cost="-17031.6" fifoPnlRealized="0" mtmPnl="72" origTradePrice="0" origTradeDate="" origTradeID="" origOrderID="0" origTransactionID="0" buySell="SELL" clearingFirmID="" ibOrderID="551923635" transactionID="2343684056" ibExecID="0002f339.657b0fca.01.01" relatedTransactionID="" brokerageOrderID="" orderReference="" volatilityOrderLink="" exchOrderId="N/A" extExecID="N/A" orderTime="20231214;092301" openDateTime="" holdingPeriodDateTime="" whenRealized="" whenReopened="" levelOfDetail="EXECUTION" changeInPrice="0" changeInQuantity="0" orderType="" traderID="" isAPIOrder="N" accruedInt="0" serialNumber="" deliveryType="" commodityType="" fineness="0.0" weight="0.0" />

Extract me here all information: ISIN, WKN, NAME, TICKER, SHARES, AMOUNT, CURRENCY (Optional Quote feed provider...like Yahoo or others)

2. CMDTY <Trade accountId="xxx" acctAlias="xxx" model="" currency="USD" fxRateToBase="0.92283" assetCategory="CMDTY" subCategory="" symbol="XAUUSD" description="London Gold" conid="69067924" securityID="" securityIDType="" cusip="" isin="" figi="" listingExchange="" underlyingConid="" underlyingSymbol="" underlyingSecurityID="" underlyingListingExchange="" issuer="" issuerCountryCode="" tradeID="645650244" multiplier="1" relatedTradeID="" strike="" reportDate="20231204" expiry="" dateTime="20231203;182345" putCall="" tradeDate="20231204" principalAdjustFactor="" settleDateTarget="20231206" transactionType="ExchTrade" exchange="UBSFX" quantity="-2" tradePrice="2115.3" tradeMoney="-4230.6" proceeds="4230.6" taxes="0" ibCommission="-1" ibCommissionCurrency="USD" netCash="4229.6" closePrice="2029.41" openCloseIndicator="C" notes="" cost="-3840.62" fifoPnlRealized="388.98" mtmPnl="171.78" origTradePrice="0" origTradeDate="" origTradeID="" origOrderID="0" origTransactionID="0" buySell="SELL" clearingFirmID="" ibOrderID="546728661" transactionID="2316076756" ibExecID="0002f339.656d0e06.01.01" relatedTransactionID="" brokerageOrderID="" orderReference="" volatilityOrderLink="" exchOrderId="N/A" extExecID="N/A" orderTime="20231203;182345" openDateTime="" holdingPeriodDateTime="" whenRealized="" whenReopened="" levelOfDetail="EXECUTION" changeInPrice="0" changeInQuantity="0" orderType="" traderID="" isAPIOrder="N" accruedInt="0" serialNumber="" deliveryType="" commodityType="" fineness="0.0" weight="0.0" />

Extract me here all information: ISIN, WKN, NAME, TICKER, SHARES, AMOUNT, CURRENCY (Optional Quote feed provider...like Yahoo or others)

Regards Alex

rektosaure commented 8 months ago

I took the time to try to understand how PP works with Interactive Brokers and FlexQuery exports.

I noticed that:

Here are the extracts you requested:

FUT: ISIN: n/a WKN: n/a NAME: description="DAX 15MAR24" TICKER: symbol="FDXS 20240315 M" SHARES: quantity="-1" * multiplier="1" AMOUNT: proceeds="17032" + ibCommission="-0.4" CURRENCY: currency="EUR" Quote Feed on Yahoo: ^GDAXI

CMDTY: ISIN: n/a WKN: n/a NAME: description="London Gold" TICKER: symbol="XAUUSD" SHARES: quantity="-2" * multiplier="1" AMOUNT: proceeds="4230.6" + ibCommission="-1" CURRENCY: currency="USD" Quote Feed on Yahoo: ^GC=F

Once again, it seems that the existing setup already works well since commodities "CMDTY" and futures "FUT" are treated the same way as stocks "STK". We could just add them in IBFlexStatementExtractorResult.

However, I haven't fully understood how forex (assetCategory="CASH") trades are processed in PP.