search

ds2-lab / infinicache

InfiniCache: A cost-effective memory cache that is built atop ephemeral serverless functions (USENIX FAST'20)
https://ds2-lab.github.io/infinicache/
MIT License
251 stars 21 forks source link

ONLY AWS? KNATIVE MIGRATION POSSIBLE? #3

Closed neboduus closed 2 years ago

neboduus commented 4 years ago

Hi all, I appreciate your effort in developing InfiniCache, and I wanted to kindly ask the authors what do they think about the migration of such serverless-based distributed in memory cache to a more portable environment such as Knative, which allows you to deploy serverless workloads using K8s. I am evaluating the possibility of implementing this idea and I would like to know your opinion if possible. Thanks in advance for any possible insight you can give me

tddg commented 4 years ago

Porting InfiniCache to open-source serverless platforms is definitely doable, with less hassles and glitches. One thing that makes such an implementation relatively easier is you essentially gain the "server" part, where a Knative function could run as a short-lived server that can accept and serve inbound connections. Whereas in Lambda-based InfiniCache, a significant portion of the code is for handling and walking around these Lambda-inherent limitations.

Another thing is the applications. InfiniCache sitting on public clouds could save money for the cloud tenants who opt to use the InfiniCache service. Likewise, an open-source implementation would demand some killer apps, which makes such an in-memory cache perfectly fit. We are also looking for application scenarios like this. Any comments or suggestions would be much appreciated towards this angle.

neboduus commented 4 years ago

Dear @tddg and and other infinicache developers/authors. Thanks a lot for the usefull suggestions. I am really interested to start such migration, For this reason I am analyzing the implementation and I came up with few considerations and many doubts about what the specific packages do. First of all I would like to tell to you what I understood and I would like to kindly ask if would be possible for you to correct what I missunderstood and please complete some information if uncomplete.

Implementation Description

The implementation is written in Go and is composed by a module github.com/mason-leap-lab/infinicache which contains 10 packages. The project go.mod file shows that it uses 13 external libraries, of which 5 indirect.

The directly imported libraries are:

The packages which compose this implementation of infinicache are:

Take into consideration that the above is just a draft which will be subject to modifications.

Until now this is my understanding of the implementation. Given that there is no documentation, I tryied to infer the above description by reading the code and matching it with the original paper. I honestly say that this is enough high level overview about how this implementation was written.

I want to kindly ask now some specific questions about the coontaining files:

I anticipate thanks for any possible insight you could give me. Of course, given that I work for implementing infinicache concepts over an opensource serverless workload management framework such as Knative, we could collaborate by exchanging informations using this thread and I can offer my knowledge as well. Thanks and bests

wangaoone commented 4 years ago

Hi @neboduus. Thanks for your interest in InfiniCache!

I appreciate the detailed specification for each file in our repo from you. this is very helpful for others to understand the src code.

As for your question,

Yes, I believe porting InfiniCache to open-source serverless platforms is workable. I am also very interested in how InfiniCache performs on an open-source serverless framework. Since the container start-up time in Knative may different from AWS Lambda. On the other side, you may not need to maintain the short Warm-up interval. But this is the idea just occurs to my mind, let's see what Infinicache will happen on Knative : )

Thanks for your work!

neboduus commented 4 years ago

Dear all!

Thanks to you for the effort for designing such innovative distributed system. I editedd the description above and if you think it might be correct, you could add it to the readme file.

I would take advantage of this thread and ask you other things during my research if for you is ok.

Thanks for any answer you could give me, your opinion is really important for me.

tddg commented 4 years ago

For your own cloud, it would be much easier IMO to handle small-object-intensive workloads, as you could allocate more concurrency or parallelism for handling bursts of small object requests within few cache instances. Yes, this limitation would not apply, to some extent.

I haven't put much thought on Q3. However, to calculate the cost for tenants' resource usage in an open deployment, it really depends. I believe you could use similar pricing model as AWS Lambda. Or, at least, you could map the same problem to priority-based multi-tenant resource sharing. This would be an interesting research question to explore.

wangaoone commented 4 years ago

Hi, for the Q3, since all the cache nodes are under AWS. The costs of the InfiniCache deployment are from AWS Lambdas. We collect log information from AWS.

neboduus commented 4 years ago

Thanks to all! I will come back with observations about this topic when I will start deploying in Knative,

zhangjyr commented 4 years ago

About lifetime

Lifetime is used to control the T_backup. while T_warm is controlled by the proxy.

infinicache-master/lambda/lifetime/lifetime.go - Backup interval controller. infinicache-master/lambda/lifetime/session.go - Invocation and bill duration controller. infinicache-master/lambda/lifetime/timeout.go - This file contains basic timer based operations that support billing duration control.

neboduus commented 4 years ago

Hi guys, apologize my stupid question: but why on the toy client app, after the GET action, I cannot see the effective value that I SET previously? How should I reconstruct it

zhangjyr commented 4 years ago

The full function signature of EcGet is: EcGet(key, size) (reqId, io.ReadCloser, success) To read data, you can simply use the second returned value, noted you must Close it even you didn't read anything. I will update the example later.

neboduus commented 4 years ago

Hi everybody, if you don't mind I would have some questions for you:

zhangjyr commented 4 years ago

That is the difference between the Lambda and Knative. For the lambda, we invoke it by calling AWS api. There is no in-bound connection allowed within the Lambda runtime. And for Knative, you can't invoke a pod, only to scale out to have more pods, and then the function logic within the pod may be invoked. In another word, a more comparable subject of the Lambda function will be the pod itself.

As for the code, we use redeo as a reverse server, which means the server on the Lambda will reuse the connection the Lambda established as a client to the proxy, and serve the request issued by the proxy. You're welcome to make connections in the way suits Knative.

neboduus commented 4 years ago

Guys thanks a lot for your suggestions. They were useful and gave me many insights about how to solve my issues.

I am happy to inform you that I have just finalized, what I called InfiniCache Lazy Migration over Knative and K8s.

Briefly I created a container for the proxy and one for the node. Then I made some modifications to the proxy and on the node in order to allow them to trigger the connection initialization using a little http server. Basically I have a GKE cluster where I installed Knative and where I am able to deploy the proxy and expose it as a K8s service and also deploy the lambda nodes as Knative serverless services/workloads. In the end I was also able to run the client example with a successful get/set.

My fork lacks of documentation now but you can have a look at the repo here if you want.

I would like to kindly ask you if would be possible to open some kind of glitter channel where we could discuss further details about your implementation. Or if I can use this issue thread to interact with you when I do not understand something. However I would appreciate any kind of possibility and channel to remain in contact with you while my work evolves.

Thanks and bests

zhangjyr commented 4 years ago

You are welcome. I will definitely take a look at code once I finished the project in hand. I guess it may be late in May. I'm thrilled we finally have a k8s based version.

neboduus commented 4 years ago

I dedicated a branch to the Lazy Migration over K8s. You can see it here. Moreover I have added some docs in proxy/docs and also a README about how it should be installed.

neboduus commented 4 years ago

Hi Guys! I am back with some new questions for my research.

Let suppose I would like to optimize infinicache to work with small objects (min 100 B, max 500 KB, just choose one). Assume a scenario where another service X is using infinicache to GET and SET always the same amount of data (min 10 KB, max 5 MB, just choose one) for each request it processes.

From the paper is clear that when the number of requests/sec exceeds 86, the pay-per-request model increases the costs too much compared to Elasticache cost which is constant.

1) Given that the objects are so small, could we reduce the number of Erasure Code shards in order to reduce the number of lambdas invocations and therefore increase the access rate without loosing the economical benefit or much performance?

Basically I am supposing that if with a RS code of 10+1 the maximum access rate before overshooting the economical benefit is 86 req/s. Reducing that number to 4+2, 5+1 (or even less if possible) will not deteriorate much the performance for so small objects but will decrease the economical cost for accessing the lambdas and therefore increase the admissible access rate.

2) Do you have any other idea about how we could optimize increase the admissible access rate?

I am supposing that the number of contacted lambdas from 11 to 6, the economical benefit would be of 2x. But I do not know how much the performance could be damaged. However I would like to be able to increase this number more, for the defined scenario.

Thanks for any possible idea

zhangjyr commented 4 years ago

Depending on the characteristic of the workload, I think we may use different EC configurations for different sizes of objects. For small objects, it will practically fallback to P+1 replicas by using EC configuration like 1+P, which may be more cost effective.

neboduus commented 4 years ago

Hi guys, can I ask why when the client detects failed chunks and tries to recover using func (c *Client) recover() does a SET() of the failed chunks? I really do not understand. Some comments say that we use a fixed lambdaID = 0. But I can't find where Infinicache discriminates on this value.

neboduus commented 4 years ago

Hello! Sorry if i am bombing you with questions, I hope to not bother you.

I also want to be able to GET more than only 1 chunk from 1 cache node (through the same connection), or maybe none. However the number of chunks I may want is dynamic.

I want to do this because I am implementing an alternative mechanism to EC which is based on replication (I want to see if it works better for small objects), but for every GET operation I need the client to select which lambda to contact therefore how many connections to open and how many chunks to download from every connection. Therefore it should be able to know where the chunks are and then ask for data. Therefore I need the client to dynamically select which lambda to talk with.

I would like to kindly ask what you think about moving the map that records which chunk on which lambda is stored, from the proxy into the client. This would allow the client to automatically select the perfect number of lambdas and stream all needed chunks.

Do you think this is possible? Do you have any suggestion regarding that or how to do it?

Thanks, and bests

zhangjyr commented 4 years ago

Hi guys, can I ask why when the client detects failed chunks and tries to recover using func (c *Client) recover() does a SET() of the failed chunks? I really do not understand. Some comments say that we use a fixed lambdaID = 0. But I can't find where Infinicache discriminates on this value.

Sorry for the late response.

Because this type of error can be fixed ubiquitously by the erasure coding algorithm at the client side. Because the proxy as a meta server has the information of the lambda the missing chunk is located, there will be no need to pass lambdaID anymore.

zhangjyr commented 4 years ago

The proxy will transform lambdaID passed by the client to real lambdaID based on multiple factors such as:

  1. Load balancing
  2. Data eviction

However, the client has solo freedom to choose EC configuration, such as 1+2 for replication (you may need to implement your own EC Encoder), or N+0 for no fault tolerance (Implemented as DummyEncoder). But you do need to create another client instance in the current implementation. I believe it is possible to implement a new client compatible with all these scenarios.

Does this answer your question?

neboduus commented 4 years ago

Hi guys, can I ask why when the client detects failed chunks and tries to recover using func (c *Client) recover() does a SET() of the failed chunks? I really do not understand. Some comments say that we use a fixed lambdaID = 0. But I can't find where Infinicache discriminates on this value.

Sorry for the late response.

Because this type of error can be fixed ubiquitously by the erasure coding algorithm at the client side. Because the proxy as a meta server has the information of the lambda the missing chunk is located, there will be no need to pass lambdaID anymore.

Thanks for the answer. Now I understand. So the decode ends recomputing the lost shards and therefore we can reset them.

neboduus commented 4 years ago

The proxy will transform lambdaID passed by the client to real lambdaID based on multiple factors such as:

1. Load balancing

2. Data eviction

However, the client has solo freedom to choose EC configuration, such as 1+2 for replication (you may need to implement your own EC Encoder), or N+0 for no fault tolerance (Implemented as DummyEncoder). But you do need to create another client instance in the current implementation. I believe it is possible to implement a new client compatible with all these scenarios.

Does this answer your question?

Thanks again. This makes sense. But how could the client dynamically decide how many connections to open if the number of chunks I may want is dynamic as well?

I will try to explain better the use case:

Suppose that each CHUNK Infinicache stores in the nodes are Key-Value pairs their self. because the data is too small (each value is at most 10 KB, sometimes just 60 B), therefore (assumption) it doesn't make sense to EC encode them.

So instead of EC encoding we replicate the chunks (each chunk stores a Key-Value Pair) across a number of nodes for fault tolerance and for load balance.

Assume you have the following Key-Value Pairs Groups

Group 1:

Group 2:

Group 3:

Each Group has a different replication factor, because they range from Frequently Accessed Data (high Replication Factor to allow load balancing) to Not So Frequently Accessed Data (so we can have a lower Replication Factor).

Moreover when the Replication Factor decreases (and therefore also the Access Frequency to that group) the number of Key-Value Pairs increases.

Assume also that these keys have some kind of relationship, which imply the following:

Which means that whenever I try to GET a Key-Value pair from Group 1, I always need 1 Key-Value Pair from Group 2 and 1 Key-Value Pair from Group 3.

For this reason we may abstract the concept and say that Group 1, 2 and 3 contain Key-Value Pairs which belong to the same Entity. This Entity is represented by a High-Level-Key.

The High-Level-Key would be the one which creates the relationship between the Groups and allows us to select to right proxy.

So now I would like to implement the following operations: 1.

SET (High-Level-Key, 
          {
          Group1: { <K1, V1> }, 
          Group2: { <K2, V2>, <K3, V3> }, 
          Group3: { <K4, V4>, <K5, V5>, <K6, V6>, <K7, V7> }
          })

Assuming that the Cache Pool is composed by the following lambdas {λa, λb, λc, ...., λz} And given the above defined Replication Factors per Group. The previous SET Operation should distribute the data in the following random 5 Cache Nodes, more or less in the following way:

λx: { <K1, V1>, <K2, V2>, <K3, V3>, <K4, V4>, <K5, V5>, <K6, V6>, <K7, V7> } λy: { <K1, V1> } λz: { <K1, V1>, <K2, V2>, <K3, V3>, <K4, V4>, <K5, V5>, <K6, V6>, <K7, V7> } λr: { <K1, V1>, <K2, V2>, <K3, V3> } λp: { <K1, V1>, <K2, V2>, <K3, V3>, <K4, V4>, <K5, V5>, <K6, V6>, <K7, V7> }

As you can see the distribution respects the Replication Factors.

2) The second operation is obviously the GET which should be able to select some or all Key-Value Pairs of a High-Level-Key. However our assumption says that we should be able to get at least 1 Key-Value Pair per High-Level-Key. Therefore our generic GET example may be:

GET (High-Level-Key, 
          {
          Group1: { <K1, V1> }, 
          Group2: { <K2, V2>}, 
          Group3: { <K4, V4> }
          })

The GET operation should have a interface similar to SET. But what should be interesting about the GET operation is that based on the Access Frequency to the cache nodes, the client should be able to decide:

This would allow to download all Key-Value Pairs specified by the GET operation either (A) from the same node (e.g. when the Access Frequency to the entire system is low) (B) or distribute the chunks requests when the nodes which contain all the keys I am interested in are over loaded

So my questions are

Forgive me if I was too long, but I would really want to know what do you think about this mechanism. Thanks and have a nice day

neboduus commented 4 years ago

If you allow me I would also want to ask you, how do you deal with concurrent SET operations. In a sense that, how do you ensure that concurrent SET operations on the same object do not corrupt the data.

I imagine that, given that the objects are EC encoded, 2 concurrent SET operations on the same object, will produce different chunks if the data is different in the 2 operations (e.g. data has changed). That's why I think that 2 concurrent operations could corrupt the data if they arrive in different orders to lambdas.

zhangjyr commented 4 years ago

I can first answer the short one. Theoretically, the value in the infinicache is unmodifiable, we should use version mechanism to deal with potential conflict (current version didn't implement this, but the experiments are based on this assumption).

zhangjyr commented 4 years ago

On the longer question. The bottom line is that the client will not need to know the real cache node the data stored. I must point out that the lambdaID the client select is not the real cache node the data will be stored. The information is stored in the proxy and is not accessible to the client. On getting data, the client specifies the chunkIDs it requires, the proxy will lookup for the real cache nodes and return data.

For set/get keys require different number of connections, note that the request is really about a chunk instead of a key, we can simply implement the batch operation of multiple chunks, and use the number of connections that equal to the key which requires most connections. The request for each connection can include different number of chunks (chunks are identified by the chunkID instead of the LambdaID).

neboduus commented 4 years ago

Hi Guys, I do not understand something from the implementation. Basically when you send a SET Request from Proxy to Lambda you add a Request to the Commands channel of an instance in this way:

p.group.Instance(lambdaDest).C() <- &types.Request{
    Id:           types.Id{connId, reqId, chunkId},
    Cmd:          strings.ToLower(c.Name),
    Key:          chunkKey,
    BodyStream:   bodyStream,
    ChanResponse: client.Responses(),
    EnableCollector: true
}

But then when you Prepare the Request for SET, you Write req.Body, which has not been set previously nowhere. Basically you do this:

func (req *Request) PrepareForSet(w *resp.RequestWriter) {
    w.WriteMultiBulkSize(6)
    w.WriteBulkString(req.Cmd)
    w.WriteBulkString(strconv.Itoa(req.Id.ConnId))
    w.WriteBulkString(req.Id.ReqId)
    w.WriteBulkString(req.Id.ChunkId)
    w.WriteBulkString(req.Key)
    if req.Body != nil {
        w.WriteBulk(req.Body)
    }
    req.w = w
}

Then my answer is: How and where is beeing req.Body set? PLEASE this question is very IMPORTANT.

And how do you think is the best way to send multiple Byte objects with the same request writer?

I was thinking to simply send a parameter which says how many Byte objects there are in the request and then simply read them in sequence.

Thanks in advance

zhangjyr commented 4 years ago

Hi, the answer is simple. We used req.BodyStream to send body instead of req.Body. The req.BodyStream was been streamed directly to the destination in req.Flush(). Notice the CopyBulk() call, this was where streaming happened.

Sending multiple byte objects as stream is a little complicated, you may have to write special reader and writer to stream streams one by one. However, write multiple byte objects is nothing different from write multiple strings.

neboduus commented 4 years ago

Hi guys! I was able to implement the idea that I mentioned in this previous post (https://github.com/mason-leap-lab/infinicache/issues/3#issuecomment-645915093) . So, I am transferring more than 1 chunk at the time using the same connection.

I am able to do that in both directions: SET and GET (proved by debug) but obviously I have a problem. It is related to the GET op:

Given that the chunks that I am transferring are so small (few Bytes), instead of streaming them, I am passing them as Byte Arguments. Therefore, for example on SET I do something like this in the client:

func (c *Client) multiKeySet(){
        ......
        w.WriteBulkString(pairs)
    for i := 0; i < pairs; i++ {
        var pair = keyValuePairs[i]
        w.WriteBulkString(pair.Key)
        w.WriteBulk(pair.Value)
    }

And I do similarly in order to transfer them from the proxy to the Lambda.

My problem arrives when the Client tries to receive data back from the Lambda during a GET op.

In the proxy I am able to receive and print the data received from the Lambda.

For example here I have created a handler for my MultiKeyGet operation, on receiving the MultiKeyGet Ack from Lambda:

func (conn *Connection) mkGetHandler(start time.Time)  {
    conn.log.Debug("mkGET from lambda %d", conn.instance.id)

    // Exhaust all values to keep protocol aligned.
    connId, _ := conn.r.ReadBulkString()
    reqId, _ := conn.r.ReadBulkString()
    chunkId, _ := conn.r.ReadBulkString()
    lowLevelKeysN, _ := conn.r.ReadInt()

    lowLevelKeyPairs := make(map[string][]byte)
    for i:=0;i<int(lowLevelKeysN);i++{
        lowLevelKey, _ := conn.r.ReadBulkString()
        var lowLevelValue []byte
        lowLevelValue, _ = conn.r.ReadBulk(lowLevelValue)
        lowLevelKeyPairs[lowLevelKey] = lowLevelValue
    }
    conn.log.Debug("received: %v", lowLevelKeyPairs)

      .....

      rsp := &types.Response{Cmd: "mkget"}
    rsp.Id.ConnId, _ = strconv.Atoi(connId)
    rsp.Id.ReqId = reqId
    rsp.Id.ChunkId = chunkId
    rsp.LowLevelKeyValuePairs = lowLevelKeyPairs

    conn.log.Debug("mkGOT %v, confirmed.", rsp.Id)
    if req, ok := conn.SetResponse(rsp); !ok {
            .......

And I am actually able to print the data received. So my GET op behaves correctly.

As you can see above I have modified the types.Response struct to add a field that I want to use instead of Body or BodyStreams. But actually I am not able to send them back to the client.

What I have done then was to add this in the `Response.Flush()" :

func (rsp *Response) Flush() error {
    if rsp.w == nil {
        return errors.New("Writer for response not set.")
    }
    w := rsp.w
    rsp.w = nil

    if rsp.BodyStream != nil {
        if err := w.CopyBulk(rsp.BodyStream, rsp.BodyStream.Len()); err != nil {
            return err
        }
    }

    l := len(rsp.LowLevelKeyValuePairs)
    if l > 0 {
        w.AppendInt(int64(l))
        for k, v := range rsp.LowLevelKeyValuePairs {
            w.AppendBulkString(k)
            w.AppendBulk(v)
        }
    }

        return w.Flush()

But this does not seem to work. On the client if I simply do lowLevelKeyValuePairsN _ = c.Conns[addr][i].R.ReadInt() it always reads 0 (zero).

Please do you have any insight about that? I want to precise again that I am sure that the data gets stuck in the proxy, once received from the lambda. So the short question would be:

How do I transfer them to the client in an By Argument Style?

Thanks and bests

zhangjyr commented 4 years ago

Did you call w.Flush() at the end of the Response.Flush?

neboduus commented 4 years ago

Hey, thanks for the answer. Yes sure, is just omited from the shown code. I edited it

neboduus commented 4 years ago

Yo in the end I managed. Was a mistake of mine. Thanks

neboduus commented 4 years ago

I can first answer the short one. Theoretically, the value in the infinicache is unmodifiable, we should use version mechanism to deal with potential conflict (current version didn't implement this, but the experiments are based on this assumption).

Hi Guys! I'd like to kindly ask some clarification about this answer if possible. I am wondering how is testing possible without being able to modify the values inside it?

The only way that can come to my mind is to test the 2 operations separately.

For the SET operation we can simply implement many concurrent clients which only SET values and therefore analyze the performance.

For the GET operation I am thinking to populate first the store with preset kv pairs and then launch many concurrent clients which perform GET operations against Infinicache.

I would like to ask if this is what you had in mind?

And also I would like to know how do you think is the better way to launch concurrent clients? Is ok to connect them against the same proxy? What about make them running on the same machine?

Thanks in advance for any possible answer and bests

zhangjyr commented 4 years ago

Do do you mean performance testing? We did that by following ways:

  1. Set different keys with one/multiple connection(s)
  2. Get different keys with one/multiple connection(s)
  3. Run real workload and analyse logged requests.

Since our paper targeted large objects and each client involved multiple connections, it can easily saturated the bandwidth of the machine. So our experiments deployed one client on each machine, and proxies were co-located with clients to form a cluster.

neboduus commented 4 years ago

Thank you for the answer.

1)When you run the real workload, dont you have concurrent GET and SET? And they may be related to same key as well. So if you can t modify the values, how did you actually managed to simulate the real workload? 2)Do you think is ok to collocate more than 1 client with the same proxy?

zhangjyr commented 4 years ago

Using version technique, updates of a key can be regarded as a key with different versions, which are equal to values with different keys. This transformation can be completed in the proxy, so that we can ensure objects are immutable. In the other hand, concurrent SETs is safe if we implements version control.

Yes, colocating multiple clients is supported, if you care more about throughput instead of latency.

neboduus commented 4 years ago

Hi! On your suggestion, I have created 2-proxy environment with 20 nodes each and I collocated 3 clients on each proxy in the following way:

func main() {
    var wg sync.WaitGroup
    for i:=0; i<3; i++{
        wg.Add(1)
        go test(i, &wg)
    }
    wg.Wait()
}

func test(i int, wg *sync.WaitGroup){
    defer wg.Done()
    var addrList = "10.4.0.100:6378,10.4.0.100:6378"
    // initial object with random value

    // parse server address
    addrArr := strings.Split(addrList, ",")

    // initial new ecRedis client
    cli := client.NewClient(10, 2, 32, 3)

    // start dial and PUT/GET
    cli.Dial(addrArr)
    var data [][3]client.KVSetGroup

    // ToDo: Replace with a channel
    var setStats []float64
    //var getStats []float64

    for k:=0; k<10000; k++{
        d := cli.GenerateSetData(5)
        data = append(data, d)
        key := fmt.Sprintf("HighLevelKey-%d", k)
        if _, stats, ok := cli.MkSet(key, d); !ok {
            log.Println("Failed to mkSET", i, key)
        }else{
            setStats = append(setStats, stats)
            log.Println("Successfull mkSET", i, key)
        }
    }

But at some point, after cca 3000 mkSET ops, a panic is thrown:

panic: runtime error: racy use of timers

goroutine 197 [running]:
time.startTimer(0xc000108968)
        /usr/local/go/src/runtime/time.go:114 +0x2b
time.(*Timer).Reset(0xc000108960, 0xdf8475800, 0xc0004efd00)
        /usr/local/go/src/time/sleep.go:127 +0x81
github.com/neboduus/infinicache/proxy/proxy/lambdastore.(*Instance).warmUp(0xc0002b91d0)
        /project/src/proxy/proxy/lambdastore/instance.go:499 +0x6a
github.com/neboduus/infinicache/proxy/proxy/lambdastore.(*Instance).flagValidated(0xc0002b91d0, 0xc000504300, 0x0)
        /project/src/proxy/proxy/lambdastore/instance.go:340 +0x7e
github.com/neboduus/infinicache/proxy/proxy/lambdastore.(*Connection).pongHandler(0xc000504300)
        /project/src/proxy/proxy/lambdastore/connection.go:242 +0x1cc
github.com/neboduus/infinicache/proxy/proxy/lambdastore.(*Connection).ServeLambda(0xc000504300)
        /project/src/proxy/proxy/lambdastore/connection.go:146 +0x76b
created by github.com/neboduus/infinicache/proxy/proxy/server.(*Proxy).Serve
        /project/src/proxy/proxy/server/proxy.go:81 +0x45

Do you have any isights about why this is happening? Thanks!

zhangjyr commented 4 years ago

It said the timer resetting was being called from multiple go-routings. So we may need to put the reset of the timer in a mutex block.

neboduus commented 4 years ago

Actually adding a mutex on line 499 seems to solve the issue. I d like to ask also:

I know that the proxy acts as a bridge between the 2 functions. And that the backup function is a replica of the node. But I cannot find the portion of code where this backup function is triggered. As happens for example for the "main" node in this function func (ins *Instance) triggerLambdaLocked(warmUp bool) {...}.

zhangjyr commented 4 years ago

The function to invoke the backup is the in the lambda side. Specifically, it is in the /lambda/migrator/client.go, line 82.