search

couchbase / sync_gateway

Manages access and synchronization between Couchbase Lite and Couchbase Server
https://www.couchbase.com/products/sync-gateway
Other
446 stars 138 forks source link

Bucket name with percent symbol is not handled correctly in config file #691

Closed lichenyang2 closed 9 years ago

lichenyang2 commented 9 years ago

According to http://docs.couchbase.com/admin/admin/UI/ui-data-buckets.html Couchbase server bucket name can contain characters in range A-Z, a-z, 0–9 as well as underscore, period, dash and percent symbols.

When the bucket name has a percent symbol in it, sync_gateway convert it to %2525 before searching for the bucket.

When there is a bucket with name of "test_%E-.5", and the config file has:

            "server": "http://172.23.100.65:8091/",
            "bucket": "test_%E-.5",

Sync_gateway would fail to start with error: 

16:48:03.459607 FATAL: Error opening database: 502 Unable to connect to server: HTTP error 404 Object Not Found getting "http://172.23.100.65:8091/pools/default/buckets/test_%2525E-.5?bucket_uuid=ddba0b7c7a41e0adcf664e22ebd571c4": Requested resource not found.
 -- rest.RunServer() at config.go:436

When there is a bucket with name of "test_%.-E5", and the config file has:

            "server": "http://172.23.100.65:8091/",
            "bucket": "test_%.-E5",

Sync_gateway would fail to start with error: 

16:03:09.459276 FATAL: Error opening database: 502 Unable to connect to server: HTTP error 404 Object Not Found getting "http://172.23.100.65:8091/pools/default/buckets/test_%2525.-E5?bucket_uuid=be0f6556a5bce55c73143a19f1fbb479": Requested resource not found.
ajres commented 9 years ago

The root cause of this issue is that Couchbase Server returns bucket URL's which are escaped. go-couchbase does not Unescape the URL's before trying to use them to connect to the bucket.

Here are the HTTP request/response flows when Sync Gateway attempts to connect to a bucket named as in the examples above.

http://localhost:8091/pools
{"pools":[{"name":"default","uri":"/pools/default?uuid=21dbcd118ebfb06643936a91dc790f57","streamingUri":"/poolsStreaming/default?uuid=21dbcd118ebfb06643936a91dc790f57"}],"isAdminCreds":false,"isROAdminCreds":false,"isEnterprise":false,"settings":{"maxParallelIndexers":"/settings/maxParallelIndexers?uuid=21dbcd118ebfb06643936a91dc790f57","viewUpdateDaemon":"/settings/viewUpdateDaemon?uuid=21dbcd118ebfb06643936a91dc790f57"},"uuid":"21dbcd118ebfb06643936a91dc790f57","implementationVersion":"3.0.1-1444-rel-community","componentsVersion":{"lhttpc":"1.3.0","os_mon":"2.2.14","public_key":"0.21","asn1":"2.0.4","couch":"2.1.1r-432-gc2af28d","kernel":"2.16.4","syntax_tools":"1.6.12","xmerl":"1.3.5","ale":"3.0.1-1444-rel-community","couch_set_view":"2.1.1r-432-gc2af28d","compiler":"4.9.4","inets":"5.9.7","mapreduce":"1.0.0","couch_index_merger":"2.1.1r-432-gc2af28d","ns_server":"3.0.1-1444-rel-community","oauth":"7d85d3ef","crypto":"3.2","ssl":"5.3.2","sasl":"2.3.4","couch_view_parser":"1.0.0","mochiweb":"2.4.2","stdlib":"1.19.4"}}
http://localhost:8091/pools/default?uuid=21dbcd118ebfb06643936a91dc790f57
{"storageTotals":{"ram":{"total":14532096000,"quotaTotal":6897532928,"quotaUsed":343932928,"used":13864841216,"usedByData":7335272,"quotaUsedPerNode":343932928,"quotaTotalPerNode":6897532928},"hdd":{"total":248391270400,"quotaTotal":248391270400,"used":243423444992,"usedByData":9780680,"free":4967825408}},"serverGroupsUri":"/pools/default/serverGroups?v=52184775","name":"default","alerts":["Approaching full disk warning. Usage of disk \"/\" on node \"127.0.0.1\" is around 97%."],"alertsSilenceURL":"/controller/resetAlerts?token=3&uuid=21dbcd118ebfb06643936a91dc790f57","nodes":[{"systemStats":{"cpu_utilization_rate":4.779874213836478,"swap_total":536870912,"swap_used":29097984,"mem_total":17179869184,"mem_free":6481235968},"interestingStats":{"cmd_get":0,"couch_docs_actual_disk_size":8908597,"couch_docs_data_size":8307264,"couch_views_actual_disk_size":872083,"couch_views_data_size":779638,"curr_items":7138,"curr_items_tot":7138,"ep_bg_fetched":0,"get_hits":0,"mem_used":7335272,"ops":0,"vb_replica_curr_items":0},"uptime":"84288","memoryTotal":17179869184,"memoryFree":6481235968,"mcdMemoryReserved":13107,"mcdMemoryAllocated":13107,"couchApiBase":"http://127.0.0.1:8092/","clusterMembership":"active","recoveryType":"none","status":"healthy","otpNode":"ns_1@127.0.0.1","thisNode":true,"hostname":"127.0.0.1:8091","clusterCompatibility":196608,"version":"3.0.1-1444-rel-community","os":"x86_64-apple-darwin10.8.0","ports":{"httpsMgmt":18091,"httpsCAPI":18092,"proxy":11211,"direct":11210}}],"buckets":{"uri":"/pools/default/buckets?v=55449572&uuid=21dbcd118ebfb06643936a91dc790f57","terseBucketsBase":"/pools/default/b/","terseStreamingBucketsBase":"/pools/default/bs/"},"remoteClusters":{"uri":"/pools/default/remoteClusters?uuid=21dbcd118ebfb06643936a91dc790f57","validateURI":"/pools/default/remoteClusters?just_validate=1"},"controllers":{"addNode":{"uri":"/controller/addNode?uuid=21dbcd118ebfb06643936a91dc790f57"},"rebalance":{"uri":"/controller/rebalance?uuid=21dbcd118ebfb06643936a91dc790f57"},"failOver":{"uri":"/controller/failOver?uuid=21dbcd118ebfb06643936a91dc790f57"},"startGracefulFailover":{"uri":"/controller/startGracefulFailover?uuid=21dbcd118ebfb06643936a91dc790f57"},"reAddNode":{"uri":"/controller/reAddNode?uuid=21dbcd118ebfb06643936a91dc790f57"},"reFailOver":{"uri":"/controller/reFailOver?uuid=21dbcd118ebfb06643936a91dc790f57"},"ejectNode":{"uri":"/controller/ejectNode?uuid=21dbcd118ebfb06643936a91dc790f57"},"setRecoveryType":{"uri":"/controller/setRecoveryType?uuid=21dbcd118ebfb06643936a91dc790f57"},"setAutoCompaction":{"uri":"/controller/setAutoCompaction?uuid=21dbcd118ebfb06643936a91dc790f57","validateURI":"/controller/setAutoCompaction?just_validate=1"},"clusterLogsCollection":{"startURI":"/controller/startLogsCollection?uuid=21dbcd118ebfb06643936a91dc790f57","cancelURI":"/controller/cancelLogsCollection?uuid=21dbcd118ebfb06643936a91dc790f57"},"replication":{"createURI":"/controller/createReplication?uuid=21dbcd118ebfb06643936a91dc790f57","validateURI":"/controller/createReplication?just_validate=1"},"setFastWarmup":{"uri":"/controller/setFastWarmup?uuid=21dbcd118ebfb06643936a91dc790f57","validateURI":"/controller/setFastWarmup?just_validate=1"}},"rebalanceStatus":"none","rebalanceProgressUri":"/pools/default/rebalanceProgress","stopRebalanceUri":"/controller/stopRebalance?uuid=21dbcd118ebfb06643936a91dc790f57","nodeStatusesUri":"/nodeStatuses","maxBucketCount":10,"autoCompactionSettings":{"parallelDBAndViewCompaction":false,"databaseFragmentationThreshold":{"percentage":30,"size":"undefined"},"viewFragmentationThreshold":{"percentage":30,"size":"undefined"}},"fastWarmupSettings":{"fastWarmupEnabled":true,"minMemoryThreshold":10,"minItemsThreshold":10},"tasks":{"uri":"/pools/default/tasks?v=23780908"},"visualSettingsUri":"/internalSettings/visual?v=7111573","counters":{}}
http://localhost:8091/pools/default/buckets?v=55449572&uuid=21dbcd118ebfb06643936a91dc790f57
[{"name":"kitchensync","bucketType":"membase","authType":"sasl","saslPassword":"","proxyPort":0,"replicaIndex":false,"uri":"/pools/default/buckets/kitchensync?bucket_uuid=e3b588fc0a681275a7fdb982b07c0751","streamingUri":"/pools/default/bucketsStreaming/kitchensync?bucket_uuid=e3b588fc0a681275a7fdb982b07c0751","localRandomKeyUri":"/pools/default/buckets/kitchensync/localRandomKey","controllers":{"compactAll":"/pools/default/buckets/kitchensync/controller/compactBucket","compactDB":"/pools/default/buckets/default/controller/compactDatabases","purgeDeletes":"/pools/default/buckets/kitchensync/controller/unsafePurgeBucket","startRecovery":"/pools/default/buckets/kitchensync/controller/startRecovery"},"nodes":[{"couchApiBaseHTTPS":"https://127.0.0.1:18092/kitchensync%2Be3b588fc0a681275a7fdb982b07c0751","couchApiBase":"http://127.0.0.1:8092/kitchensync%2Be3b588fc0a681275a7fdb982b07c0751","systemStats":{"cpu_utilization_rate":4.779874213836478,"swap_total":536870912,"swap_used":29097984,"mem_total":17179869184,"mem_free":6481235968},"interestingStats":{"cmd_get":0,"couch_docs_actual_disk_size":8908597,"couch_docs_data_size":8307264,"couch_views_actual_disk_size":872083,"couch_views_data_size":779638,"curr_items":7138,"curr_items_tot":7138,"ep_bg_fetched":0,"get_hits":0,"mem_used":7335272,"ops":0,"vb_replica_curr_items":0},"uptime":"84288","memoryTotal":17179869184,"memoryFree":6481235968,"mcdMemoryReserved":13107,"mcdMemoryAllocated":13107,"replication":0,"clusterMembership":"active","recoveryType":"none","status":"healthy","otpNode":"ns_1@127.0.0.1","thisNode":true,"hostname":"127.0.0.1:8091","clusterCompatibility":196608,"version":"3.0.1-1444-rel-community","os":"x86_64-apple-darwin10.8.0","ports":{"httpsMgmt":18091,"httpsCAPI":18092,"proxy":11211,"direct":11210}}],"stats":{"uri":"/pools/default/buckets/kitchensync/stats","directoryURI":"/pools/default/buckets/kitchensync/statsDirectory","nodeStatsListURI":"/pools/default/buckets/kitchensync/nodes"},"ddocs":{"uri":"/pools/default/buckets/kitchensync/ddocs"},"nodeLocator":"vbucket","fastWarmupSettings":false,"autoCompactionSettings":false,"uuid":"e3b588fc0a681275a7fdb982b07c0751","vBucketServerMap":{"hashAlgorithm":"CRC","numReplicas":1,"serverList":["127.0.0.1:11210"],"vBucketMap":[[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1]]},"replicaNumber":1,"threadsNumber":3,"quota":{"ram":209715200,"rawRAM":209715200},"basicStats":{"quotaPercentUsed":2.328121185302734,"opsPerSec":0,"diskFetches":0,"itemCount":7138,"diskUsed":9238082,"dataUsed":7780032,"memUsed":4882424},"evictionPolicy":"valueOnly","bucketCapabilitiesVer":"","bucketCapabilities":["cbhello","touch","couchapi","cccp","xdcrCheckpointing","nodesExt"]},{"name":"test_%E-.5","bucketType":"membase","authType":"sasl","saslPassword":"","proxyPort":0,"replicaIndex":false,"uri":"/pools/default/buckets/test_%25E-.5?bucket_uuid=0a7bcb44051f2d5d62c51c7f03dcf3f3","streamingUri":"/pools/default/bucketsStreaming/test_%25E-.5?bucket_uuid=0a7bcb44051f2d5d62c51c7f03dcf3f3","localRandomKeyUri":"/pools/default/buckets/test_%25E-.5/localRandomKey","controllers":{"compactAll":"/pools/default/buckets/test_%25E-.5/controller/compactBucket","compactDB":"/pools/default/buckets/default/controller/compactDatabases","purgeDeletes":"/pools/default/buckets/test_%25E-.5/controller/unsafePurgeBucket","startRecovery":"/pools/default/buckets/test_%25E-.5/controller/startRecovery"},"nodes":[{"couchApiBaseHTTPS":"https://127.0.0.1:18092/test_%25E-.5%2B0a7bcb44051f2d5d62c51c7f03dcf3f3","couchApiBase":"http://127.0.0.1:8092/test_%25E-.5%2B0a7bcb44051f2d5d62c51c7f03dcf3f3","systemStats":{"cpu_utilization_rate":4.779874213836478,"swap_total":536870912,"swap_used":29097984,"mem_total":17179869184,"mem_free":6481235968},"interestingStats":{"cmd_get":0,"couch_docs_actual_disk_size":8908597,"couch_docs_data_size":8307264,"couch_views_actual_disk_size":872083,"couch_views_data_size":779638,"curr_items":7138,"curr_items_tot":7138,"ep_bg_fetched":0,"get_hits":0,"mem_used":7335272,"ops":0,"vb_replica_curr_items":0},"uptime":"84288","memoryTotal":17179869184,"memoryFree":6481235968,"mcdMemoryReserved":13107,"mcdMemoryAllocated":13107,"replication":0,"clusterMembership":"active","recoveryType":"none","status":"healthy","otpNode":"ns_1@127.0.0.1","thisNode":true,"hostname":"127.0.0.1:8091","clusterCompatibility":196608,"version":"3.0.1-1444-rel-community","os":"x86_64-apple-darwin10.8.0","ports":{"httpsMgmt":18091,"httpsCAPI":18092,"proxy":11211,"direct":11210}}],"stats":{"uri":"/pools/default/buckets/test_%25E-.5/stats","directoryURI":"/pools/default/buckets/test_%25E-.5/statsDirectory","nodeStatsListURI":"/pools/default/buckets/test_%25E-.5/nodes"},"ddocs":{"uri":"/pools/default/buckets/test_%25E-.5/ddocs"},"nodeLocator":"vbucket","fastWarmupSettings":false,"autoCompactionSettings":false,"uuid":"0a7bcb44051f2d5d62c51c7f03dcf3f3","vBucketServerMap":{"hashAlgorithm":"CRC","numReplicas":1,"serverList":["127.0.0.1:11210"],"vBucketMap":[[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1]]},"replicaNumber":1,"threadsNumber":3,"quota":{"ram":134217728,"rawRAM":134217728},"basicStats":{"quotaPercentUsed":1.827514171600342,"opsPerSec":0,"diskFetches":0,"itemCount":0,"diskUsed":542598,"dataUsed":527232,"memUsed":2452848},"evictionPolicy":"valueOnly","bucketCapabilitiesVer":"","bucketCapabilities":["cbhello","touch","couchapi","cccp","xdcrCheckpointing","nodesExt"]}]

In the last response the bucket name is returned unescaped:

"name":"test_%E-.5"

But the resource URL's are returned escaped, e.g.:

"compactAll":"/pools/default/buckets/test_%25E-.5/controller/compactBucket"

This URL needs to be unescaped by go-couchbase before attempting to send a further request to the URL.

ajres commented 9 years ago

Raised PR against go-couchbase

ajres commented 9 years ago

After cherry picking the following commits to master :

https://github.com/couchbase/sync_gateway/commit/191adbce6c25dce35c065449a6dc1d3bbaff9163

https://github.com/couchbase/sync_gateway/commit/2651970f72bc371ef0aa502532156c2e656c811e

Connecting to buckets with % in their names is successful.

Closing this ticket out.