search

Graphcool / graphcool-framework

Apache License 2.0
1.77k stars 131 forks source link

Graphcool-framework export does not seem to work far away clusters #456

Open marktani opened 6 years ago

marktani commented 6 years ago

Issue by Kisepro Wednesday Jan 10, 2018 at 17:25 GMT Originally opened as https://github.com/graphcool/prisma/issues/1570

Hello,

I'm trying to export data from an asian cluster but it's not working on any of them. However it's working on every eu server.

PS C:\Project\myBjjGit\project\server> graphcool-framework export -e myData.zip -t asiaTest
Downloading nodes... √
 !    The download of nodes failed. You may get fragmented data. Request ID: eu-west-1:simple:cjc9bxi2qlnv20136vk3688ra

Downloading lists... 173ms
Downloading relations... √
 !    The download of relations failed. You may get fragmented data. Request ID: eu-west-1:simple:cjc9bxifxldwx0102djs8dtg5

Zipping export...
Zipping export... 39ms
Exported service to myData.zip
You can import it to a new service with
  $ graphcool-framework import --source myData.zip --target target-name
PS C:\Project\myBjjGit\project\server> graphcool-framework export -e myData.zip -t euTest
Downloading nodes... 210ms
Downloading lists... 157ms
Downloading relations... 26718ms
Zipping export...
Zipping export... 50ms
Exported service to myData.zip
You can import it to a new service with
  $ graphcool-framework import --source myData.zip --target target-name

To reproduce, I just use the same schema and deployed it on eu and asian server.

Maybe related to #1565

`-- graphcool-framework@0.11.5

mattfysh commented 6 years ago

@marktani this issue is also affecting shared-eu-west-1

mattfysh commented 6 years ago

any idea when this may be fixed? I know the export function is not high priority, but for us we're blocked from upgrading to Framework as we'd need to take a backup of our production data before upgrading.

marktani commented 6 years ago

Hey @mattfysh, after further investigation, this seems to be a problem when you're trying to export data "across regions". So if you're trying to export data from a project in shared-eu-west-1, try to send the export request from a european IP.

This might help you work around this issue until it's fixed from our side 🙂

mattfysh commented 6 years ago

Hey @marktani - this worked! Using NordVPN, able to connect via UK and export. thanks ✌️

tsdexter commented 6 years ago

@marktani do you know if this will be fixed in the near term or would it be wise to configure a VPN?

alternatively, is there a way to clone an existing baas to another region. ie: clone an account from eu to us?

mattfysh commented 6 years ago

@marktani I just ran another data backup (can never be too careful!) and this time "Downloading lists..." failed, even after using NordVPN to connect via the UK.

Downloading nodes... 2460ms
Downloading lists... ✔
 ▸    The download of lists failed. You may get fragmented data. Request ID: eu-west-1:simple:cjekua381n6rg0143wm19a113

Downloading relations... 1030ms
Zipping export... ⣻
Zipping export... 116ms
Exported service to export-2018-03-10T03:57:29.967Z.zip

Do you know if this is okay to ignore? e.g. would I still be able to restore (import) this data to production or another GC target?

byeblogs commented 6 years ago

my question just this : @marktani how to trying this "send the export request from a european IP"..??

this is just info about my log : My problems exactly same, when try to export nodes from shared-eu-west-1.. i already try using raw or using graphcool export -t dev.. the zip already downloaded but,

in log :

Downloading lists... 1313ms Downloading relations... ✔ ▸ The download of relations failed. You may get fragmented data. Request ID: ap-northeast-1:simple:cjekt7sw1xe5p0127wmnqyw2c Zipping export... Zipping export... 45ms Exported service to export-2018-03-10T03:27:40.669Z.zip You can import it to a new service with $ graphcool-framework import --source export-2018-03-10T03:27:40.669Z.zip --target target-name

when i open zip folder, there is no info about my nodes etc..

tsdexter commented 6 years ago

Rather than use a VPN I've managed to export my data using Google Cloud

  1. create an F1-micro compute engine instance in Google Cloud with NodeJS (https://console.cloud.google.com/launcher/details/click-to-deploy-images/nodejs) in the region your GraphCool project is in (ie: eu-west).

  2. SSH into the machine with the google web based terminal

    screen shot 2018-04-09 at 10 12 02 pm

  3. run npm install graphcool-framework -g

  4. run sudo nano .graphcoolrc

  5. paste in your .graphcoolrc file from your project

  6. run (replacing your token and target)

    GRAPHCOOL_PLATFORM_TOKEN=__YOUR_PLATFORM_TOKEN__ gcf export -e backup.zip -t __TARGET_FROM_GRAPHCOOLRC__

  7. wait

  8. If you don't know what folder you're in, run pwd it should be something like /home/username

  9. Choose Download file from the top-right context menu

    screen shot 2018-04-09 at 10 16 22 pm

  10. Type /home/username/backup.zip (using your username and filename you set in step 6

...

profit?

baerrach commented 6 years ago

I did essentially the same thing with AWS EC2 instance in the same region as GraphCool.

mattfysh commented 6 years ago

@marktani any thoughts on why I'm getting "The download of lists failed" but nodes and relations are okay? thx!

calummoore commented 6 years ago

I'm experiencing this issue from a UK based IP.

Downloading nodes... 980ms
Downloading lists... ✔
 ▸    The download of lists failed. You may get fragmented data. Request ID:
 ▸    eu-west-1:simple:cjgpl4gvepcqz01714kcycdd9

Downloading relations... 238ms
Zipping export...
Zipping export... 35ms
Exported service to export-2018-05-02T20:59:28.864Z.zip
You can import it to a new service with
  $ graphcool import --source export-2018-05-02T20:59:28.864Z.zip --target target-name
baerrach commented 6 years ago

Please note this affects the "import" command too.

After dumping the fetch() call for the import the error response { requestId: 'ap-northeast-1:simple:XXXX', } points to the wrong cluster. This should be shared-eu-west-1, not ap-northeast-1

hakimelek commented 5 years ago

@marktani Any updates on this? The export is holding some of my projects :(

baerrach commented 5 years ago

Use one of the work-arounds. Either VPN or EU AWS EC2 instance.

hakimelek commented 5 years ago

@baerrach tried VPN from Europe, no luck.

baerrach commented 5 years ago

It worked for me.

I can only assume your cluster is not in the EU region.

Your .graphcoolrc file will list your targets and the cluster it belongs too.

Make sure you connect from an IP that the CDN will recognise as one close to that region.

bre7 commented 5 years ago

It could also be related with #544 instead, in which case you're out of luck since I doubt it'll be fixed

hakimelek commented 5 years ago

@baerrach I have tried to deploy it in the 3 regions: shared-eu-west-1, shared-us-west-2 and shared-ap-northeast-1, VPN from Europe, same issue. Also, whenever I try to export, I noticed that the region is always us-west-2 which is my initial region.

baerrach commented 5 years ago

@hakimelek How are you noticing what the region is when exporting? When I run my export command with DEBUG=* there is no information about regions at all.

hakimelek commented 5 years ago

@baerrach I can see that in the requestId 

WARNING: Error: The download of relations failed. You may get fragmented data. Request ID: us-west-2:simple:xxxxxxxxxxxxxxxxxxxx

Digging more into the data returned from the request, this is the data, that is causing the error. 

{ requestId: 'us-west-2:simple:xxxxxxxxxxxxxxxxxxxx',
  error: 'Whoops. Looks like an internal server error. Please contact us from the Console (https://console.graph.cool) or via email (support@graph.cool) and include your Request ID: us-west-2:simple:xxxxxxxxxxxxxxxxxxxx' }
danielnakasato commented 5 years ago

@hakimelek Any updates on this case? I am having the same errors. Tried to run an instance on EC2 (eu-west-1) but the problem still happens.

mattfysh commented 5 years ago

@danielnakasato I had to resort to writing a manual script that exports the data in NDF (https://www.graph.cool/docs/reference/data-import-and-export/normalized-data-format-teroo5uxih)

danielnakasato commented 5 years ago

@mattfysh Thanks for the answer! Just one more question... Did you manage to keep the objects IDs? I have some external services whose identifiers were created considering the user IDs...

mattfysh commented 5 years ago

Good question! The json files you create in NDF format will include existing system metadata including ID's, created/updated timestamps. As long as you import using the CLI, the target will retain the object IDs.

hakimelek commented 5 years ago

@danielnakasato I ended up just making different queries to all my database and download the data as json files then built a small script to make the migration, not the best but worked for me :/

sreuter commented 5 years ago

Save yourself from messing around with EC2 or VPN.

Just query a nameserver in the region your looking to export from... For EU you can do something like:

dig @78.35.40.149 api.graph.cool
...
;; ANSWER SECTION:
api.graph.cool.     60  IN  A   52.222.161.23
api.graph.cool.     60  IN  A   52.222.161.75
...

Then just put one of the IPs into your /etc/hosts ...

52.222.161.130 api.graph.cool

Retry your export and it will work :)

lastmjs commented 5 years ago

@sreuter I've done as you've suggested, but it doesn't work for me...I get the exact same error. When I ping api.graph.cool after updating /etc/hosts, I get the result from the correct IP address. I'm not sure what's wrong.

sreuter commented 5 years ago

@lastmjs Which region is your cluster and which ip did you put into the hosts file?

lastmjs commented 5 years ago

Nevermind, I ended up spinning up an EC2 instance in the same region and that didn't even work. Eu-west-1 I believe. I tried all of the IP addresses returned from the dig command

lastmjs commented 5 years ago

They were essentially the same as yours, 52.222.161.x

marceloogeda commented 4 years ago

I've faced the same problem. It only worked for me when I change the region on my compute engine instance in Google Cloud.

CE - us-west1-a Graphcool (shared cluster hosted) - shared-eu-west-1

Thank you @tsdexter you help me a lot. :pray: