Open PaulNewton opened 5 years ago
Hey @PaulNewton thanks for opening the issue!
A malformed response should not be returned in normal usages, I will work on making that error message more informative however you should not be receiving that message. (Which is probably why that error message sucks in the first place)
I would love it if you could download a free copy of Charles Proxy When you run it the first time it should start recording with a proxy running on port 8888.
Then you can run theme get -t=[your-id] --timeout=1m --proxy=http://127.0.0.1:8888
This will record all the requests from themekit. You can inspect the body of each requests to see how they look. If you notice anything you can let me know. Otherwise if you rightclick on the parent you will be able to export the report and send it to me tim.anema@shopify.com
Thanks for you help!
I ran ahead and assumed normal troubleshooting problems on my end including bad connection, so changed network, restarted my machine, deleted the current folders&files and started again. All files downloaded no errors, possible correlation 🤔 ? Will test again on other network along with powershell when I get a chance. @PaulNewton TODO
- Are you by any chance running a proxy or anything in-between you and Shopify currently?
there was an auth gateway so possible proxy or filter. For now I changed networks and themekit get is working without errors so possible correlation.
- Are you running powershell, cmd.exe, WSL?
Running commands in cmd.exe
- Could you try it in an alternative shell?
Haven't tried powershell yet on prior network
- Is this your first installation of themekit?
Not my first install of themekit
- Has it worked before?
Until this error Themekit has worked like a dream, thank you so much.
- Did you install using chocolatey?
I installed themekit straight from the github download
A note on finding if anythings missing doing a file compare of the admin exported theme.zip and themekit download. The export zip will insert rendered versions of css.liquid and javascript.liquid increase the file count in the zip. I've gotten so used to themekit I forgot that happened.
there was an auth gateway so possible proxy or filter. For now I changed networks and themekit get is working without errors so possible correlation.
I have seen this cause issues in the past so I can probably give you my 80~90% confidence that it was this that was interfering with themekit. I would love to make themekit more resilient to this issue, so if you run into the problem again I would love it if you could capture some of the network traffic and hopefully I can prevent these sort of issues in the future.
I'm encountering these errors, for a development store.
@tanema I've emailed you a Charles Proxy export file.
I don't believe it's a network issue for me, as themekit is working fine for other live stores.
I'm not behind a proxy.
I'm using bash, on OS X
I think my particular case these "received a malformed response" themekit errors might be because I'm trying to download a trial theme.
When I used themekit to download the default debut theme, it's working fine.
@monting I haven't had a chance to look at your proxy output yet but thankyou for that extra info. I will test with a trial theme as well. That would indicate that we are experiencing 2 problems here though
Running into this issue as well. Will try diff'ing theme downloaded via zip archive against theme download command, as well as inspecting Charles Proxy logs.
@cameroncowden and @monting if you are on OSX you can use mitmproxy.
Install it with brew install mitmproxy
run it with mitmproxy -p 5000 -w themekit_dump
then run themekit in another console with theme get -t=[id] --proxy=--proxy=http://localhost:5000
then you can exit mitmproxy by pressing q (then y for yes) and send me the themekit_dump file.
Mitmproxy is more minimal, easy to use, and easier for me to parse.
Hey Tim,
So, it appears the error doesn't occur if the proxy option is used.
hrm, that might indicate that it has something to do with ssl/https since ssl is disabled while proxying so that all traffic can be inspected. It would also explain why the logs that everyone sent me were completely unintelligible because they were still encoded for some reason. It might have something to do with this other issue: https://github.com/Shopify/themekit/issues/544
A fix for this is on master and you can try it out with themekit update --version=v1.0.1-pre
and let me know if that makes any difference.
Roger that. I'm currently seeing this error at a coffee shop and the wifi has no password. I'm not sure if I've seen this on my home wifi (wpa2 I think). Not sure if that's helpful.
I also have the theme I'm downloading in a git repo, and even when the errors is reported there doesn't appear to be anything incorrect or jumbled in the files that are updated or added. (I wouldn't know if it "skipped" one of the new files though)
I have released 1.0.1 with the fix for #544 in it. Please give it a try and see if it solves your issues.
I am getting the error on my site page: This page is temporarily unavailable because a device from your location is sending large amounts of web requests. Visitors from other locations can still view the page.
Theme kit gives me the error: This page is temporarily unavailable because a device from your location is sending large amounts of web requests. Visitors from other locations can still view the page.
The errors (which happen simultaneously so I assume its the same issue) are intermittent. Everything works fine; then it doesn't. If I CTRL-C and wait a few minutes, it starts working again. But then soon I get the same error.
I am using the Chrome extension LivePage to refresh my page since PrePros doesn't work on my system for some reason.
I've set my config.yml file to include: bucket_size: 40, refill_rate: 2, concurrency: 2, timeout: 30s.
I'm posting here because the error is the same but I don't know if the cause is the same. I tried running the Charles Proxy command as listed above but I get a response: Proxy URL detected from Configuration[http://127.0.0.1:8888] SSL Certificate Validation will be disabled! Get https://tales-of-murder-press.myshopify.com/meta.json: proxyconnect tcp: dial tcp 127.0.0.1:8888: connect: connection refused.
I don't believe I'm behind a proxy (I didn't set up anything).
I'm running Manjaro Linux (up to date). Chrome.
@flaturtha you definitely have a proxy configured if that message is happening. Check your config file or your THEMEKIT_PROXY environment variable. Also are you working in an office with other theme developers all sharing the same IP? If so it is just likely that your whole office is being throttled because of a lot of traffic.
I’m working solo in a home office so no other users. I’ll check my config and environment variables.
I am using the Chrome extension LivePage so maybe that’s the root of the proxy issue. The problem goes away when it’s turned off but of course I lose browser refresh, etc.
Thanks. I’m learning but still very much a noob so I’m not always sure where an issue comes from.
I'm having this issue all of a sudden. I've been working for a week with my theme. All of a sudden tonight while using theme watch
I refreshed and suddenly theme when nuts. Now my theme is borked, and totally using wrong theme settings. I can't upload.
1) I've tried creating a new theme entirely with theme deploy. No dice. 2) I've tried deleting all the theme files on the server, but no go.
I'm not going to lie I'm kind of freaking out. I even rolled back to a few version of my theme in git and no dice. Any ideas?
I adjusted the timeout to 10min. Still the same errors. I just tried it on different ISPs. I don't have any proxy installed. Additionally the files it complains about if I upload them with theme deploy
on just the individual file it's fine. But I can't upload or fix my theme. Something about assets.json seems bad on shopify's servers.
Here you can take a look. My theme is totally borked. It's using the wrong style, colors, and has a whole bunch of modules I normally don't have. https://0gu1633t8wuopedl-14175736.shopifypreview.com/
When I go into the shopify admin it claims some file I've never edited is broken. (I checked git history)
Thanks for any help!
@ckizer I would recommend restarting your terminal and/or your computer and try deploying again. The deploy command will upload a copy of each file to make sure the theme is fully sync so that makes sense if a single file in the theme is corrupted. If restarting does not work I would recommend trying different wifi, like your home wifi.
It may also be that Shopify was having issues at the time you were deploying, and it may be clear now.
Same error this morning.
200|227 [====================================================================>---------] 88 % [development] (assets/thin-leadin-page-5.jpg) received a malformed response from shopify, this usually indicates a problem with your connection [development] (assets/slider-cervical-2.jpg) Put https://cradle-me.myshopify.com/admin/themes/74947723367/assets.json: stream error: stream ID 197; PROTOCOL_ERROR [development] (assets/thin-leadin-page-7.jpg) Put https://cradle-me.myshopify.com/admin/themes/74947723367/assets.json: stream error: stream ID 199; PROTOCOL_ERROR [development] (assets/hero-cervical.jpg) Put https://cradle-me.myshopify.com/admin/themes/74947723367/assets.json: stream error: stream ID 201; PROTOCOL_ERROR [development] (assets/pacific.js.map) Put https://cradle-me.myshopify.com/admin/themes/74947723367/assets.json: stream error: stream ID 205; PROTOCOL_ERROR [development] (assets/animate.css) Put https://cradle-me.myshopify.com/admin/themes/74947723367/assets.json: stream error: stream ID 219; PROTOCOL_ERROR [development] (assets/thin-gray-page-2.jpg) Put https://cradle-me.myshopify.com/admin/themes/74947723367/assets.json: stream error: stream ID 213; PROTOCOL_ERROR [development] (assets/slider-boomerang-3.jpg) Put https://cradle-me.myshopify.com/admin/themes/74947723367/assets.json: stream error: stream ID 187; PROTOCOL_ERROR [development] (assets/cervical-leadin-page-6.jpg) Put https://cradle-me.myshopify.com/admin/themes/74947723367/assets.json: stream error: stream ID 191; PROTOCOL_ERROR [development] (assets/thin-leadin-page-8.jpg) received a malformed response from shopify, this usually indicates a problem with your connection
I've restarted. Tried an entirely different computer with a new checked out version of my theme @tanema
The issues started around 12:30am at Shopify. They aren't fixed still. It's def on Shopify's servers as the same thing is happening on a new computer with a brand new checked out copy.
I consider myself a decent developer, and have no idea how to proceed. I even tried looking for the assets.json file to delete.
Any of those protocol error files above if I manually deploy them it works fine deploy assets/cervical-leadin-page-6.jpg
works. So it's not the file
It seems like your issue is not the same issue as this. Yours is actually a bit lower in the stack before json is unmarshalled so I have opened a new issue to track this #659 Please follow there
I have also noticed that this might happen when shopify returns html error pages if something is wrong. I will try to add some checks to see if I can at least handle that instance.
So, is it so then that Trial Themes shoul work fine? Im having exactly this same problem for a theme trial. If I try in the same shop theme kit for an free one which is Live now it works, but for the Theme Trial it does not.
Theme trials do not let you access the themes code neither through the admin or the api AFAIK , though this seems like this should be a 403 access forbidden instead of this error.
On Wed, Apr 8, 2020, 4:33 AM oscarsan notifications@github.com wrote:
So, is it so then that Trial Themes shoul work fine? Im having exactly this same problem for a theme trial. If I try in the same shop theme kit for an free one which is Live now it works, but for the Theme Trial it does not.
— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/Shopify/themekit/issues/631#issuecomment-610827392, or unsubscribe https://github.com/notifications/unsubscribe-auth/ALIKKMMQRU2UU54LOG6U42DRLQZDXANCNFSM4HIG5EHQ .
I have solved this issue, just use a proxy server and it is works for me. This is probably happen because my IP is unstable that changed automatically in several short time
I tried using a proxy server as well but to no avail. Currently the only way I've been able to get this manageable is by re-publishing until the only errors I have are images that have no bearing on the theme functionality.
Currently my internet is pretty slow due to where I'm living at the moment but I cannot change that right now (3mb down, 1mb up).
I've also set the timeout to 10 minutes but again same errors. Also this has been with Shopify Plus as well basic Shopify plans. Running on MacOS 10.15.4 using VS Code (zsh shell) using ThemeKit 1.0.3).
Anyone found insights into this? Thanks!
@jonpennington
until the only errors I have are images that have no bearing on the theme functionality
What happens if explicitly download everything BUT images?
(double check syntax):
themekit download layout/* templates/* sections/* snippets/* config/* locales/* assets/*.liquid assets/*.js assets/*.css assets/*.scss
etc
Or making a config that ignores all image files https://shopify.github.io/themekit/ignores/
development-download-without-images:
readonly: true
ignore_files:
# https://shopify.github.io/themekit/ignores/
- "*.png" # patterns that start with * need to be quoted to have vaild yaml
# raster formats
- /\.(bmp|gif|jpg|jpeg|webp|tiff)$/
# vector formats
- /\.(svg|svgz|eps|epsi|ink|tex|pdf|ai|indd|vsd)$/
#raw & data formats
- ?\.(raw|exif|cr2|nef|orf|sr2)$/
# software container formats
- /\.(psd|pdf|ai|indd|xcf|cdr)$/
I haven't had issues with download but with publishing.
Using the ignore_files setup you had I only had issues with MP4/WebM files (I then re-ran it with those file types being ignored). On second run, I had a 71kb js file cause this issues on publishing:
(assets/ella.js) Put https://....myshopify.com/admin/themes/.../assets.json: stream error: stream ID 233; PROTOCOL_ERROR
So it definitely seemed to reduce the amount of files having issues (in this case often it's 5-7). But even when I ran it through a proxy...it seems the PUT commands get queued and at some point they must time out or hit some limitation from the server?
(assets/....js)
Probably unrelated and merely a coincidence but that js file indicates it's the same theme that first gave me these types of errors. Which at the time seemed like a combo of the wifi portal used and maybe the large images assets.
@jonpennington Anyone found insights into this? Thanks!
Unfortunately I think repeatability is the problem here, I've not seen it again since themekit was updated with #544 thanks to tanema, however I also haven't touched that theme again but I made a laundry list to go through if it happened again since it kills productivity when it happens. To rule out cases such as specific themes being culprits anyone wanting to hunt this down could try:
That at least narrows it more to the theme itself or some themekit bug and not single network oddness Make a backup export of the theme:
theme download
-> theme deploy
either over the problem theme or a new foldertheme new
to generate a theme and repeat themekit commands either over the problem theme or a new folderOkay, just to let you know I am still working on this. I just opened a PR that will give us better error outputs for this. The error will include the HTTP Status Code so that we can figure out what sort of response we are getting. I am also flushing the response body to a temporary file, so that we can examine the response. I am pretty sure that the response we are getting is an html error page (which we cannot unmarshal) so if we can see the html page, we can get what the error message is.
Since we will not be merging anything around Black Friday/Cyber Monday, I have created a prerelease with the additions for this issue included in it. You can install the prerelease by running theme update --version=v1.1.3-pre
, please share any feedback that you have on this.
Not quite sure if my issue is related to this one, but... theme download
cannot download images, css or settings_data.json.
The things that are not downloaded are not larger than 600KB.
@alexissel for the images, do you have any ignore rules in your config for the development environment?
Nope
Edit: This is with a different theme:
Context Cancelled
is worriesome. Right now I am cancelling requests with context when we get a 499 status response. I did not think that this would effect responses that have already completed. I think I might be able to make a work around for this.
I am having the same issue as above but when calling a specific theme download
command for an asset glob. For example:
theme download assets/*.woff assets/*.woff2 assets/*.otf assets/*.ttf --no-ignore -p=[pass] -s=[store] -d=src/assets
All instances fail with the same problem as above: Context Cancelled
. I have provided samples of output below.
[development] Errors encountered:
[development] error downloading assets/Gotham-Bold.woff2: could not read response body
This may mean that the request was not able to finish successfully
Http Response Status: 200
Request ID: 0bcf25df-9182-4a7d-8ff1-95fdee1bfa29
Error: context canceled
[development] error downloading assets/Gotham-BookItalic.woff2: could not read response body
This may mean that the request was not able to finish successfully
Http Response Status: 200
Request ID: e8d3f185-3010-4914-aaaf-986743d39e76
Error: context canceled
finished command with errors
[development] Errors encountered:
[development] error downloading assets/bundle.ajax-search.js: could not read response body
This may mean that the request was not able to finish successfully
Http Response Status: 200
Request ID: 7ff048f1-48c5-4e34-b5d1-7cfc28ce8aa1
Error: context canceled
finished command with errors
[development] Errors encountered:
[development] error downloading assets/Gotham-MediumItalic.woff2: could not read response body
This may mean that the request was not able to finish successfully
Http Response Status: 200
Request ID: 89ab149a-1323-4be2-b0e6-a9342db2a05d
Error: context canceled
[development] error downloading assets/Gotham-LightItalic.woff2: could not read response body
This may mean that the request was not able to finish successfully
Http Response Status: 200
Request ID: 4ebbfca3-ba9d-4f0a-b5a6-56bc22499387
Error: context canceled
[development] error downloading assets/Gotham-BookItalic.woff2: could not read response body
This may mean that the request was not able to finish successfully
Http Response Status: 200
Request ID: 1991085b-07d0-46f1-8f45-876a2fa4964b
Error: context canceled
finished command with errors
Also getting these kinds of errors on theme download
calls:
[development] Errors encountered:
[development] error downloading assets/icons.svg: could not read response body
This may mean that the request was not able to finish successfully
Http Response Status: 200
Request ID: 91d535db-7713-41a6-861a-5892e243d707
Error: context canceled
finished command with errors
After a retry it usually works, sometimes I need two retries. Has been like this for months now.
Is your feature request related to a problem? Please describe. Using version 1.0.0 windows 10 64 bit When downloading theme files, using themekit get -t=THEMEID, you may get the following message output:
And the progress count indicates all files have downloaded but some files are actually missing when checking the folders&files on the OS.
error downloading asset: received a malformed response
Which seems to be from ErrMalformedResponse Without the verbose (-v) flag after the download process finishes this may be displayed multiple times.With the verbose flag I will see the above error in between messages of "Successfully wrote X to disk" so creates an ambiguity because what does that mean when following an error?
Without the filename or a checksum system you have to download the theme zip and do a comparison if you want to soothe the file paranoia, and then copy any missing files
Describe the solution you'd like Included the relevant filename so interpretation of the message is easier. Or clarifying if a file needs to be redownloaded either in the console or in the documentation. Or provide a log of affected files to then pipe back into themekit for individual download. Or prompt user to retry missing files
Describe alternatives you've considered Alternative is to manually download theme zip from the store admin via email kinda rendering themekits download capability pointless when these errors occur.
Additional context I'm not able to reliably reproduce the same output with the -v flag or narrow down if it's specific file types, or as a problem on shopifys end or my end. The theme was created less than an hour before running command Increasing the timeout to 1minute doesn't make it go away.