search

yndx-metrika / logs_api_integration

Script for integration with Logs API
46 stars 50 forks source link

Ошибка в разборе данных #4

Open Delimer opened 7 years ago

Delimer commented 7 years ago

Пытаюсь выбрать все данные по визитам за пару дней python metrica_logs_api.py -source visits -start_date 2017-03-01 -end_date 2017-03-03

Конфиг выглядит так { "token" : "...", "counter_id": "...", "visits_fields": [ "ym:s:visitID", "ym:s:counterID", "ym:s:watchIDs", "ym:s:date", "ym:s:dateTime", "ym:s:dateTimeUTC", "ym:s:isNewUser", "ym:s:startURL", "ym:s:endURL", "ym:s:pageViews", "ym:s:visitDuration", "ym:s:bounce", "ym:s:ipAddress", "ym:s:params", "ym:s:goalsID", "ym:s:goalsSerialNumber", "ym:s:goalsDateTime", "ym:s:goalsPrice", "ym:s:goalsOrder", "ym:s:goalsCurrency", "ym:s:clientID", "ym:s:lastTrafficSource", "ym:s:lastAdvEngine", "ym:s:lastReferalSource", "ym:s:lastSearchEngineRoot", "ym:s:lastSearchEngine", "ym:s:lastSocialNetwork", "ym:s:lastSocialNetworkProfile", "ym:s:referer", "ym:s:lastDirectClickOrder", "ym:s:lastDirectBannerGroup", "ym:s:lastDirectClickBanner", "ym:s:lastDirectPhraseOrCond", "ym:s:lastDirectPlatformType", "ym:s:lastDirectPlatform", "ym:s:lastDirectConditionType", "ym:s:lastCurrencyID", "ym:s:from", "ym:s:UTMCampaign", "ym:s:UTMContent", "ym:s:UTMMedium", "ym:s:UTMSource", "ym:s:UTMTerm", "ym:s:openstatAd", "ym:s:openstatCampaign", "ym:s:openstatService", "ym:s:openstatSource", "ym:s:hasGCLID", "ym:s:regionCountry", "ym:s:regionCity", "ym:s:browserLanguage", "ym:s:browserCountry", "ym:s:clientTimeZone", "ym:s:deviceCategory", "ym:s:mobilePhone", "ym:s:mobilePhoneModel", "ym:s:operatingSystemRoot", "ym:s:operatingSystem", "ym:s:browser", "ym:s:browserMajorVersion", "ym:s:browserMinorVersion", "ym:s:browserEngine", "ym:s:browserEngineVersion1", "ym:s:browserEngineVersion2", "ym:s:browserEngineVersion3", "ym:s:browserEngineVersion4", "ym:s:cookieEnabled", "ym:s:javascriptEnabled", "ym:s:flashMajor", "ym:s:flashMinor", "ym:s:screenFormat", "ym:s:screenColors", "ym:s:screenOrientation", "ym:s:screenWidth", "ym:s:screenHeight", "ym:s:physicalScreenWidth", "ym:s:physicalScreenHeight", "ym:s:windowClientWidth", "ym:s:windowClientHeight", "ym:s:purchaseID", "ym:s:purchaseDateTime", "ym:s:purchaseAffiliation", "ym:s:purchaseRevenue", "ym:s:purchaseTax", "ym:s:purchaseShipping", "ym:s:purchaseCoupon", "ym:s:purchaseCurrency", "ym:s:purchaseProductQuantity", "ym:s:productsPurchaseID", "ym:s:productsID", "ym:s:productsName", "ym:s:productsBrand", "ym:s:productsCategory", "ym:s:productsCategory1", "ym:s:productsCategory2", "ym:s:productsCategory3", "ym:s:productsCategory4", "ym:s:productsCategory5", "ym:s:productsVariant", "ym:s:productsPosition", "ym:s:productsPrice", "ym:s:productsCurrency", "ym:s:productsCoupon", "ym:s:productsQuantity", "ym:s:impressionsURL", "ym:s:impressionsDateTime", "ym:s:impressionsProductID", "ym:s:impressionsProductName", "ym:s:impressionsProductBrand", "ym:s:impressionsProductCategory", "ym:s:impressionsProductCategory1", "ym:s:impressionsProductCategory2", "ym:s:impressionsProductCategory3", "ym:s:impressionsProductCategory4", "ym:s:impressionsProductCategory5", "ym:s:impressionsProductVariant", "ym:s:impressionsProductPrice", "ym:s:impressionsProductCurrency", "ym:s:impressionsProductCoupon", "ym:s:lastDirectClickOrderName", "ym:s:lastClickBannerGroupName", "ym:s:lastDirectClickBannerName", "ym:s:networkType" ], "hits_fields": [ "ym:pv:counterID", "ym:pv:dateTime", "ym:pv:date", "ym:pv:firstPartyCookie" ], "log_level": "INFO", "retries": 1, "retries_delay": 60, "clickhouse": { "host": "http://localhost:8123", "user": "...", "password": "...", "visits_table": "visits_all", "hits_table": "hits_all", "database": "..." } }

Полученные данные и ошибки `2017-03-03 14:42:03 MainProcess INFO ### DATA SAMPLE 2017-03-03 14:42:03 MainProcess INFO ym:s:bounce ym:s:browser ym:s:browserCountry ym:s:browserEngine ym:s:browserEngineVersion1 ym:s:browserEngineVersion2 ym:s:browserEngineVersion3 ym:s:browserEngineVersion4 ym:s:browserLanguage ym:s:browserMajorVersion ym:s:browserMinorVersion ym:s:clientID ym:s:clientTimeZone ym:s:cookieEnabled ym:s:counterID ym:s:date ym:s:dateTime ym:s:dateTimeUTC ym:s:deviceCategory ym:s:endURL ym:s:flashMajor ym:s:flashMinor ym:s:from ym:s:goalsCurrency ym:s:goalsDateTime ym:s:goalsID ym:s:goalsOrder ym:s:goalsPrice ym:s:goalsSerialNumber ym:s:hasGCLID ym:s:impressionsDateTime ym:s:impressionsProductBrand ym:s:impressionsProductCategory ym:s:impressionsProductCategory1 ym:s:impressionsProductCategory2 ym:s:impressionsProductCategory3 ym:s:impressionsProductCategory4 ym:s:impressionsProductCategory5 ym:s:impressionsProductCoupon ym:s:impressionsProductCurrency ym:s:impressionsProductID ym:s:impressionsProductName ym:s:impressionsProductPrice ym:s:impressionsProductVariant ym:s:impressionsURL ym:s:ipAddress ym:s:isNewUser ym:s:javascriptEnabled ym:s:lastAdvEngine ym:s:lastClickBannerGroupName ym:s:lastCurrencyID ym:s:lastDirectBannerGroup ym:s:lastDirectClickBanner ym:s:lastDirectClickBannerName ym:s:lastDirectClickOrder ym:s:lastDirectClickOrderName ym:s:lastDirectConditionType ym:s:lastDirectPhraseOrCond ym:s:lastDirectPlatform ym:s:lastDirectPlatformType ym:s:lastReferalSource ym:s:lastSearchEngine ym:s:lastSearchEngineRoot ym:s:lastSocialNetwork ym:s:lastSocialNetworkProfile ym:s:lastTrafficSource ym:s:mobilePhone ym:s:mobilePhoneModel ym:s:networkType ym:s:openstatAd ym:s:openstatCampaign ym:s:openstatService ym:s:openstatSource ym:s:operatingSystem ym:s:operatingSystemRoot ym:s:pageViews ym:s:params ym:s:physicalScreenHeight ym:s:physicalScreenWidth ym:s:productsBrand ym:s:productsCategory ym:s:productsCategory1 ym:s:productsCategory2 ym:s:productsCategory3 ym:s:productsCategory4 ym:s:productsCategory5 ym:s:productsCoupon ym:s:productsCurrency ym:s:productsID ym:s:productsName ym:s:productsPosition ym:s:productsPrice ym:s:productsPurchaseID ym:s:productsQuantity ym:s:productsVariant ym:s:purchaseAffiliation ym:s:purchaseCoupon ym:s:purchaseCurrency ym:s:purchaseDateTime ym:s:purchaseID ym:s:purchaseProductQuantity ym:s:purchaseRevenue ym:s:purchaseShipping ym:s:purchaseTax ym:s:referer ym:s:regionCity ym:s:regionCountry ym:s:screenColors ym:s:screenFormat ym:s:screenHeight ym:s:screenOrientation ym:s:screenWidth ym:s:startURL ym:s:UTMCampaign ym:s:UTMContent ym:s:UTMMedium ym:s:UTMSource ym:s:UTMTermym:s:visitDuration ym:s:visitID ym:s:watchIDs ym:s:windowClientHeight ym:s:windowClientWidth 0 chrome 0 WebKit 537 36 0 0 30066 56 0 1488484141656852122 180 1 421224 2017-03-02 2017-03-02 22:48:55 2017-03-02 22:48:55 1 https://... 0 0 [0] [\'2017-03-02 22:49:58\'] [14196793]...

....

2017-03-03 14:42:03 MainProcess CRITICAL Iteration #1 failed Traceback (most recent call last): File "metrica_logs_api.py", line 127, in integrate_with_logs_api(config, user_request) File "metrica_logs_api.py", line 107, in integrate_with_logs_api raise e ValueError: Code: 26, e.displayText() = DB::Exception: Cannot parse quoted string: expected opening quote: (at row 1)

Row 1: Column 0, name: Bounce, type: UInt8, parsed text: "0" Column 1, name: Browser, type: String, parsed text: "chrome" Column 2, name: BrowserCountry, type: String, parsed text: "0" Column 3, name: BrowserEngine, type: String, parsed text: "WebKit" Column 4, name: BrowserEngineVersion1, type: UInt16, parsed text: "537" Column 5, name: BrowserEngineVersion2, type: UInt16, parsed text: "36" Column 6, name: BrowserEngineVersion3, type: UInt16, parsed text: "0" Column 7, name: BrowserEngineVersion4, type: UInt16, parsed text: "0" Column 8, name: BrowserLanguage, type: String, parsed text: "30066" Column 9, name: BrowserMajorVersion, type: UInt16, parsed text: "56" Column 10, name: BrowserMinorVersion, type: UInt16, parsed text: "0" Column 11, name: ClientID, type: UInt64, parsed text: "1488484141656852122" Column 12, name: ClientTimeZone, type: Int16, parsed text: "180" Column 13, name: CookieEnabled, type: UInt8, parsed text: "1" Column 14, name: CounterID, type: UInt32, parsed text: "421224" Column 15, name: Date, type: Date, parsed text: "2017-03-02" Column 16, name: DateTime, type: DateTime, parsed text: "2017-03-02 22:48:55" Column 17, name: DateTimeUTC, type: DateTime, parsed text: "2017-03-02 22:48:55" Column 18, name: DeviceCategory, type: String, parsed text: "1" Column 19, name: EndURL, type: String, parsed text: "https://..." Column 20, name: FlashMajor, type: UInt8, parsed text: "0" Column 21, name: FlashMinor, type: UInt8, parsed text: "0" Column 22, name: From, type: String, parsed text: Column 23, name: GoalsCurrency, type: Array(String), parsed text: "["ERROR

, e.what() = DB::Exception `

Slach commented 6 years ago

Недавно формат выгрузки поправили, попробуйте еще раз