search

icatproject / icat.oaipmh

OAI-PMH implementation for ICAT
Apache License 2.0
0 stars 1 forks source link

Connection reset when maxResults is increased #12

Closed antolinos closed 4 years ago

antolinos commented 4 years ago

If I increase the maxResults to 500 in run.properties file and get all identifiers (I receive 500) and then for every identifier I get its record the server will complain with a connection reset error.

Log:

2020-03-17 22:19:59,494 INFO [http-thread-pool::http-listener-1(264)] RequestInterface - Initialized RequestInterface
2020-03-17 22:19:59,496 INFO [http-thread-pool::http-listener-1(294)] RequestInterface - Initialized RequestInterface
2020-03-17 22:19:59,497 INFO [http-thread-pool::http-listener-1(278)] RequestInterface - Initialized RequestInterface
2020-03-17 22:19:59,499 INFO [http-thread-pool::http-listener-1(133)] RequestInterface - Initialized RequestInterface
2020-03-17 22:19:59,501 INFO [http-thread-pool::http-listener-1(175)] RequestInterface - Initialized RequestInterface
2020-03-17 22:19:59,503 INFO [http-thread-pool::http-listener-1(295)] RequestInterface - Initialized RequestInterface
2020-03-17 22:19:59,504 INFO [http-thread-pool::http-listener-1(248)] RequestInterface - Initialized RequestInterface
2020-03-17 22:19:59,505 INFO [http-thread-pool::http-listener-1(262)] RequestInterface - Initialized RequestInterface
2020-03-17 22:19:59,507 INFO [http-thread-pool::http-listener-1(279)] RequestInterface - Initialized RequestInterface
2020-03-17 22:19:59,509 INFO [http-thread-pool::http-listener-1(238)] RequestInterface - Initialized RequestInterface
2020-03-17 22:19:59,511 INFO [http-thread-pool::http-listener-1(287)] RequestInterface - Initialized RequestInterface
2020-03-17 22:19:59,512 INFO [http-thread-pool::http-listener-1(250)] RequestInterface - Initialized RequestInterface
2020-03-17 22:19:59,513 INFO [http-thread-pool::http-listener-1(300)] RequestInterface - Initialized RequestInterface
2020-03-17 22:19:59,515 INFO [http-thread-pool::http-listener-1(123)] RequestInterface - Initialized RequestInterface
2020-03-17 22:19:59,516 INFO [http-thread-pool::http-listener-1(137)] RequestInterface - Initialized RequestInterface
2020-03-17 22:19:59,518 INFO [http-thread-pool::http-listener-1(247)] RequestInterface - Initialized RequestInterface
2020-03-17 22:19:59,519 INFO [http-thread-pool::http-listener-1(208)] RequestInterface - Initialized RequestInterface
2020-03-17 22:19:59,527 INFO [http-thread-pool::http-listener-1(213)] RequestInterface - Initialized RequestInterface
2020-03-17 22:19:59,528 INFO [http-thread-pool::http-listener-1(297)] RequestInterface - Initialized RequestInterface
2020-03-17 22:19:59,530 INFO [http-thread-pool::http-listener-1(242)] RequestInterface - Initialized RequestInterface
2020-03-17 22:19:59,532 INFO [http-thread-pool::http-listener-1(128)] RequestInterface - Initialized RequestInterface
2020-03-17 22:19:59,534 INFO [http-thread-pool::http-listener-1(184)] RequestInterface - Initialized RequestInterface
2020-03-17 22:19:59,536 INFO [http-thread-pool::http-listener-1(260)] RequestInterface - Initialized RequestInterface
2020-03-17 22:19:59,537 INFO [http-thread-pool::http-listener-1(256)] RequestInterface - Initialized RequestInterface
2020-03-17 22:19:59,538 INFO [http-thread-pool::http-listener-1(269)] RequestInterface - Initialized RequestInterface
2020-03-17 22:20:28,609 ERROR [http-thread-pool::http-listener-1(3)] ResponseBuilder - class java.net.SocketException Connection reset
2020-03-17 22:20:28,610 ERROR [http-thread-pool::http-listener-1(168)] ResponseBuilder - class java.net.SocketException Connection reset

Property changed (run.properties):

# The maximum number of results per page for incomplete responses
maxResults = 500

My test:

describe('OAI-PMH', () => {
    describe('/oai-pmh request', () => {
        it.each(oaipmhResource.requests, '[%s]', ['description'], function (element, next) {
       // Request the list of identifiers
            global.gRequester
            .get('/oaipmh/'+ element.requestListIdentifiers)
            .send()
            .end((err, getOAIResponse) => {     
        expect(getOAIResponse.status).to.equal(200);                            
        let identifiers = (XmlQuery(XmlReader.parseSync(getOAIResponse.text)).find('identifier'));
        let counter = 1;
        identifiers.each(identifier => {
            // Request by investigation identifier
             global.gRequester
                        .get('/oaipmh/'+ element.requestByIdentifier + identifier.children[0].value)
                        .send()
                        .end((err, getInvestigationResponse) => {
                    expect(getInvestigationResponse.status).to.equal(200);    
                    counter = counter + 1;
                    let title = (XmlQuery(XmlReader.parseSync(getInvestigationResponse.text)).find('dc:title').text());
                    // check that there are no forbidden proposal types
                    element.expected.forbiddenProposalTypes.forEach(forbiddenProposalType => {
                        expect(title.toLowerCase().startsWith(forbiddenProposalType)).to.be.false;
                    });
                    // end of tests condition
                    if (counter == identifiers.ast.length){
                        next();
                    }
            })
        });                 
            });           
        }
    )        
 });
EmilJunker commented 4 years ago

Thanks a lot for bringing this to my attention. Let me just ask a few questions for clarification.

antolinos commented 4 years ago

Dear @EmilJunker

Thanks a lot for bringing this to my attention. Let me just ask a few questions for clarification.

  • Can you confirm that in response to the initial ListIdentifiers request, you indeed receive a list of 500 identifiers?

I confirm

  • When maxResults is set to the default value of 50, does your test succeed?

Yes

I have added some logs. for the test when maxResults = 50

Server is listening...
  OAI-PMH
    /oai-pmh request
#identifiers:50
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
      ✓ [Check that non-public investigations are exposed] (2826ms)

Server is closed
MongoDB is stopped.

  1 passing (12s)
  • Do you know how many of the GetRecord requests succeed before the connection is reset? In other words, what is the final value of the counter variable?

when maxResults = 500 

Server is listening...
  OAI-PMH
    /oai-pmh request
#identifiers:500
2
3
      1) [Check that non-public investigations are exposed]
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42

43
44
Server is closed
MongoDB is stopped.

  0 passing (44s)
  1 failing

  1) OAI-PMH
       /oai-pmh request
         [Check that non-public investigations are exposed]:

      Uncaught AssertionError: expected 500 to equal 200
      + expected - actual

      -500
      +200

      at /users/demariaa/Software/elogbook/icat+/test/routes/test.oaipmh.routes.js:27:49
      at Test.Request.callback (node_modules/superagent/lib/node/index.js:716:12)
      at /users/demariaa/Software/elogbook/icat+/node_modules/superagent/lib/node/index.js:916:18
      at IncomingMessage.<anonymous> (node_modules/superagent/lib/node/parsers/json.js:19:7)
      at endReadableNT (_stream_readable.js:1206:12)
      at processTicksAndRejections (internal/process/task_queues.js:84:21)

I would say that ICAT can not deal with 500 requests at the same time.

antolinos commented 4 years ago

To confirm my hypothesis when adding some sleep time between calls:

Server is listening...
  OAI-PMH
    /oai-pmh request
#identifiers:500
1 (oai:icatplus.esrf.fr:inv/13171260) with sleep 100
2 (oai:icatplus.esrf.fr:inv/13175377) with sleep 200
3 (oai:icatplus.esrf.fr:inv/13178078) with sleep 300
4 (oai:icatplus.esrf.fr:inv/13449074) with sleep 400
5 (oai:icatplus.esrf.fr:inv/15527191) with sleep 500
6 (oai:icatplus.esrf.fr:inv/18443302) with sleep 600
7 (oai:icatplus.esrf.fr:inv/20306689) with sleep 700
8 (oai:icatplus.esrf.fr:inv/21481600) with sleep 800
9 (oai:icatplus.esrf.fr:inv/24211970) with sleep 900
10 (oai:icatplus.esrf.fr:inv/24212102) with sleep 1000
11 (oai:icatplus.esrf.fr:inv/24303857) with sleep 1100
12 (oai:icatplus.esrf.fr:inv/25934013) with sleep 1200
13 (oai:icatplus.esrf.fr:inv/27341687) with sleep 1300
14 (oai:icatplus.esrf.fr:inv/30150919) with sleep 1400
15 (oai:icatplus.esrf.fr:inv/34497655) with sleep 1500
16 (oai:icatplus.esrf.fr:inv/34876560) with sleep 1600
17 (oai:icatplus.esrf.fr:inv/38942075) with sleep 1700
18 (oai:icatplus.esrf.fr:inv/39148064) with sleep 1800
19 (oai:icatplus.esrf.fr:inv/39153692) with sleep 1900
20 (oai:icatplus.esrf.fr:inv/39438105) with sleep 2000
21 (oai:icatplus.esrf.fr:inv/47877949) with sleep 2100
22 (oai:icatplus.esrf.fr:inv/52749136) with sleep 2200
23 (oai:icatplus.esrf.fr:inv/52776499) with sleep 2300
24 (oai:icatplus.esrf.fr:inv/52898290) with sleep 2400
25 (oai:icatplus.esrf.fr:inv/53827377) with sleep 2500
26 (oai:icatplus.esrf.fr:inv/53832340) with sleep 2600
27 (oai:icatplus.esrf.fr:inv/54308426) with sleep 2700
28 (oai:icatplus.esrf.fr:inv/54384876) with sleep 2800
29 (oai:icatplus.esrf.fr:inv/54639288) with sleep 2900
30 (oai:icatplus.esrf.fr:inv/54642592) with sleep 3000
31 (oai:icatplus.esrf.fr:inv/54672194) with sleep 3100
32 (oai:icatplus.esrf.fr:inv/54792166) with sleep 3200
33 (oai:icatplus.esrf.fr:inv/55108289) with sleep 3300
34 (oai:icatplus.esrf.fr:inv/55384807) with sleep 3400
35 (oai:icatplus.esrf.fr:inv/55416273) with sleep 3500
36 (oai:icatplus.esrf.fr:inv/55558035) with sleep 3600
37 (oai:icatplus.esrf.fr:inv/55833529) with sleep 3700
38 (oai:icatplus.esrf.fr:inv/56717200) with sleep 3800
39 (oai:icatplus.esrf.fr:inv/56717247) with sleep 3900
40 (oai:icatplus.esrf.fr:inv/57176032) with sleep 4000
41 (oai:icatplus.esrf.fr:inv/57502481) with sleep 4100
42 (oai:icatplus.esrf.fr:inv/57504067) with sleep 4200
43 (oai:icatplus.esrf.fr:inv/57726980) with sleep 4300
44 (oai:icatplus.esrf.fr:inv/58077227) with sleep 4400
45 (oai:icatplus.esrf.fr:inv/58494317) with sleep 4500
46 (oai:icatplus.esrf.fr:inv/58581398) with sleep 4600
47 (oai:icatplus.esrf.fr:inv/58784441) with sleep 4700
48 (oai:icatplus.esrf.fr:inv/58831478) with sleep 4800
49 (oai:icatplus.esrf.fr:inv/58957084) with sleep 4900
50 (oai:icatplus.esrf.fr:inv/59284909) with sleep 5000
51 (oai:icatplus.esrf.fr:inv/59321374) with sleep 5100
52 (oai:icatplus.esrf.fr:inv/59508766) with sleep 5200
53 (oai:icatplus.esrf.fr:inv/59848051) with sleep 5300
54 (oai:icatplus.esrf.fr:inv/59883644) with sleep 5400
55 (oai:icatplus.esrf.fr:inv/60225627) with sleep 5500
56 (oai:icatplus.esrf.fr:inv/60226184) with sleep 5600
57 (oai:icatplus.esrf.fr:inv/60682942) with sleep 5700
58 (oai:icatplus.esrf.fr:inv/60977488) with sleep 5800
59 (oai:icatplus.esrf.fr:inv/61665128) with sleep 5900
60 (oai:icatplus.esrf.fr:inv/61783686) with sleep 6000
61 (oai:icatplus.esrf.fr:inv/61873366) with sleep 6100
62 (oai:icatplus.esrf.fr:inv/61877104) with sleep 6200
63 (oai:icatplus.esrf.fr:inv/62054746) with sleep 6300
64 (oai:icatplus.esrf.fr:inv/62119185) with sleep 6400
65 (oai:icatplus.esrf.fr:inv/62119779) with sleep 6500
66 (oai:icatplus.esrf.fr:inv/66521977) with sleep 6600
67 (oai:icatplus.esrf.fr:inv/66943547) with sleep 6700
68 (oai:icatplus.esrf.fr:inv/67090769) with sleep 6800
69 (oai:icatplus.esrf.fr:inv/67159411) with sleep 6900
70 (oai:icatplus.esrf.fr:inv/67542531) with sleep 7000
71 (oai:icatplus.esrf.fr:inv/68260307) with sleep 7100
72 (oai:icatplus.esrf.fr:inv/68682878) with sleep 7200
73 (oai:icatplus.esrf.fr:inv/68688215) with sleep 7300
74 (oai:icatplus.esrf.fr:inv/69376545) with sleep 7400
75 (oai:icatplus.esrf.fr:inv/69596120) with sleep 7500
76 (oai:icatplus.esrf.fr:inv/70356463) with sleep 7600
77 (oai:icatplus.esrf.fr:inv/71096239) with sleep 7700
78 (oai:icatplus.esrf.fr:inv/71391221) with sleep 7800
79 (oai:icatplus.esrf.fr:inv/71559333) with sleep 7900
80 (oai:icatplus.esrf.fr:inv/72804369) with sleep 8000
81 (oai:icatplus.esrf.fr:inv/73115504) with sleep 8100
82 (oai:icatplus.esrf.fr:inv/74255164) with sleep 8200
83 (oai:icatplus.esrf.fr:inv/74746123) with sleep 8300
84 (oai:icatplus.esrf.fr:inv/74836812) with sleep 8400
85 (oai:icatplus.esrf.fr:inv/74854051) with sleep 8500
86 (oai:icatplus.esrf.fr:inv/76355704) with sleep 8600
87 (oai:icatplus.esrf.fr:inv/76990955) with sleep 8700
88 (oai:icatplus.esrf.fr:inv/77484279) with sleep 8800
89 (oai:icatplus.esrf.fr:inv/79263824) with sleep 8900
90 (oai:icatplus.esrf.fr:inv/81910232) with sleep 9000
91 (oai:icatplus.esrf.fr:inv/82159689) with sleep 9100
92 (oai:icatplus.esrf.fr:inv/83826208) with sleep 9200
93 (oai:icatplus.esrf.fr:inv/84010164) with sleep 9300
94 (oai:icatplus.esrf.fr:inv/84145492) with sleep 9400
95 (oai:icatplus.esrf.fr:inv/84149394) with sleep 9500
96 (oai:icatplus.esrf.fr:inv/85392633) with sleep 9600
97 (oai:icatplus.esrf.fr:inv/85858239) with sleep 9700
98 (oai:icatplus.esrf.fr:inv/86039273) with sleep 9800
99 (oai:icatplus.esrf.fr:inv/86410123) with sleep 9900
100 (oai:icatplus.esrf.fr:inv/86533633) with sleep 10000
101 (oai:icatplus.esrf.fr:inv/86896510) with sleep 10100
102 (oai:icatplus.esrf.fr:inv/87410954) with sleep 10200
103 (oai:icatplus.esrf.fr:inv/87416237) with sleep 10300
104 (oai:icatplus.esrf.fr:inv/87561975) with sleep 10400
105 (oai:icatplus.esrf.fr:inv/88836881) with sleep 10500
106 (oai:icatplus.esrf.fr:inv/89193924) with sleep 10600
107 (oai:icatplus.esrf.fr:inv/90375186) with sleep 10700
108 (oai:icatplus.esrf.fr:inv/90375370) with sleep 10800
109 (oai:icatplus.esrf.fr:inv/90378880) with sleep 10900
110 (oai:icatplus.esrf.fr:inv/90531358) with sleep 11000
111 (oai:icatplus.esrf.fr:inv/90610071) with sleep 11100
112 (oai:icatplus.esrf.fr:inv/90632078) with sleep 11200
113 (oai:icatplus.esrf.fr:inv/90632248) with sleep 11300
114 (oai:icatplus.esrf.fr:inv/90632739) with sleep 11400
115 (oai:icatplus.esrf.fr:inv/90633157) with sleep 11500
116 (oai:icatplus.esrf.fr:inv/90633392) with sleep 11600
117 (oai:icatplus.esrf.fr:inv/91076597) with sleep 11700
118 (oai:icatplus.esrf.fr:inv/91076856) with sleep 11800
119 (oai:icatplus.esrf.fr:inv/91421585) with sleep 11900
120 (oai:icatplus.esrf.fr:inv/91585925) with sleep 12000
121 (oai:icatplus.esrf.fr:inv/91586319) with sleep 12100
122 (oai:icatplus.esrf.fr:inv/91586611) with sleep 12200
123 (oai:icatplus.esrf.fr:inv/91656957) with sleep 12300
124 (oai:icatplus.esrf.fr:inv/91732236) with sleep 12400
125 (oai:icatplus.esrf.fr:inv/91732296) with sleep 12500
126 (oai:icatplus.esrf.fr:inv/91822219) with sleep 12600
127 (oai:icatplus.esrf.fr:inv/91836717) with sleep 12700
128 (oai:icatplus.esrf.fr:inv/91846265) with sleep 12800
129 (oai:icatplus.esrf.fr:inv/91846670) with sleep 12900
130 (oai:icatplus.esrf.fr:inv/91877587) with sleep 13000
131 (oai:icatplus.esrf.fr:inv/92150081) with sleep 13100
132 (oai:icatplus.esrf.fr:inv/92161891) with sleep 13200
133 (oai:icatplus.esrf.fr:inv/92161916) with sleep 13300
134 (oai:icatplus.esrf.fr:inv/92162610) with sleep 13400
135 (oai:icatplus.esrf.fr:inv/92162619) with sleep 13500
136 (oai:icatplus.esrf.fr:inv/92177754) with sleep 13600
137 (oai:icatplus.esrf.fr:inv/92202780) with sleep 13700
138 (oai:icatplus.esrf.fr:inv/92219443) with sleep 13800
139 (oai:icatplus.esrf.fr:inv/92349871) with sleep 13900
140 (oai:icatplus.esrf.fr:inv/92352178) with sleep 14000
141 (oai:icatplus.esrf.fr:inv/93253723) with sleep 14100
142 (oai:icatplus.esrf.fr:inv/93592827) with sleep 14200
143 (oai:icatplus.esrf.fr:inv/93597902) with sleep 14300
144 (oai:icatplus.esrf.fr:inv/93598391) with sleep 14400
145 (oai:icatplus.esrf.fr:inv/93644946) with sleep 14500
146 (oai:icatplus.esrf.fr:inv/93841705) with sleep 14600
147 (oai:icatplus.esrf.fr:inv/93862624) with sleep 14700
148 (oai:icatplus.esrf.fr:inv/94190939) with sleep 14800
149 (oai:icatplus.esrf.fr:inv/94201225) with sleep 14900
150 (oai:icatplus.esrf.fr:inv/94212235) with sleep 15000
151 (oai:icatplus.esrf.fr:inv/94374108) with sleep 15100
152 (oai:icatplus.esrf.fr:inv/94520246) with sleep 15200
153 (oai:icatplus.esrf.fr:inv/94529374) with sleep 15300
154 (oai:icatplus.esrf.fr:inv/94535688) with sleep 15400
155 (oai:icatplus.esrf.fr:inv/94636543) with sleep 15500
156 (oai:icatplus.esrf.fr:inv/94638761) with sleep 15600
157 (oai:icatplus.esrf.fr:inv/94756894) with sleep 15700
158 (oai:icatplus.esrf.fr:inv/94920677) with sleep 15800
159 (oai:icatplus.esrf.fr:inv/94929323) with sleep 15900
160 (oai:icatplus.esrf.fr:inv/94944488) with sleep 16000
161 (oai:icatplus.esrf.fr:inv/95212225) with sleep 16100
162 (oai:icatplus.esrf.fr:inv/95212808) with sleep 16200
163 (oai:icatplus.esrf.fr:inv/95217558) with sleep 16300
164 (oai:icatplus.esrf.fr:inv/95282956) with sleep 16400
165 (oai:icatplus.esrf.fr:inv/95455645) with sleep 16500
166 (oai:icatplus.esrf.fr:inv/95600596) with sleep 16600
167 (oai:icatplus.esrf.fr:inv/95601432) with sleep 16700
168 (oai:icatplus.esrf.fr:inv/95608373) with sleep 16800
169 (oai:icatplus.esrf.fr:inv/95608574) with sleep 16900
170 (oai:icatplus.esrf.fr:inv/95902814) with sleep 17000
171 (oai:icatplus.esrf.fr:inv/95938495) with sleep 17100
172 (oai:icatplus.esrf.fr:inv/95941505) with sleep 17200
173 (oai:icatplus.esrf.fr:inv/95945234) with sleep 17300
174 (oai:icatplus.esrf.fr:inv/95948696) with sleep 17400
175 (oai:icatplus.esrf.fr:inv/96318785) with sleep 17500
176 (oai:icatplus.esrf.fr:inv/96499898) with sleep 17600
177 (oai:icatplus.esrf.fr:inv/96614611) with sleep 17700
178 (oai:icatplus.esrf.fr:inv/96614964) with sleep 17800
179 (oai:icatplus.esrf.fr:inv/96822620) with sleep 17900
180 (oai:icatplus.esrf.fr:inv/96822720) with sleep 18000
181 (oai:icatplus.esrf.fr:inv/96822926) with sleep 18100
182 (oai:icatplus.esrf.fr:inv/96823783) with sleep 18200
183 (oai:icatplus.esrf.fr:inv/96823860) with sleep 18300
184 (oai:icatplus.esrf.fr:inv/97059468) with sleep 18400
185 (oai:icatplus.esrf.fr:inv/97062972) with sleep 18500
186 (oai:icatplus.esrf.fr:inv/97284682) with sleep 18600
187 (oai:icatplus.esrf.fr:inv/97308600) with sleep 18700
188 (oai:icatplus.esrf.fr:inv/97349939) with sleep 18800
189 (oai:icatplus.esrf.fr:inv/97543424) with sleep 18900
190 (oai:icatplus.esrf.fr:inv/97560425) with sleep 19000
191 (oai:icatplus.esrf.fr:inv/97579051) with sleep 19100
192 (oai:icatplus.esrf.fr:inv/97692825) with sleep 19200
193 (oai:icatplus.esrf.fr:inv/97799000) with sleep 19300
194 (oai:icatplus.esrf.fr:inv/98484251) with sleep 19400
195 (oai:icatplus.esrf.fr:inv/98516694) with sleep 19500
196 (oai:icatplus.esrf.fr:inv/99036937) with sleep 19600
197 (oai:icatplus.esrf.fr:inv/99051610) with sleep 19700
198 (oai:icatplus.esrf.fr:inv/99054302) with sleep 19800
199 (oai:icatplus.esrf.fr:inv/99268493) with sleep 19900
200 (oai:icatplus.esrf.fr:inv/100129017) with sleep 20000
201 (oai:icatplus.esrf.fr:inv/100329643) with sleep 20100
202 (oai:icatplus.esrf.fr:inv/100389184) with sleep 20200
203 (oai:icatplus.esrf.fr:inv/100444815) with sleep 20300
204 (oai:icatplus.esrf.fr:inv/100483209) with sleep 20400
205 (oai:icatplus.esrf.fr:inv/100748218) with sleep 20500
206 (oai:icatplus.esrf.fr:inv/101113846) with sleep 20600
207 (oai:icatplus.esrf.fr:inv/101127303) with sleep 20700
208 (oai:icatplus.esrf.fr:inv/101569525) with sleep 20800
209 (oai:icatplus.esrf.fr:inv/101823522) with sleep 20900
210 (oai:icatplus.esrf.fr:inv/101843223) with sleep 21000
211 (oai:icatplus.esrf.fr:inv/101872286) with sleep 21100
212 (oai:icatplus.esrf.fr:inv/101905463) with sleep 21200
213 (oai:icatplus.esrf.fr:inv/102373644) with sleep 21300
214 (oai:icatplus.esrf.fr:inv/102440920) with sleep 21400
215 (oai:icatplus.esrf.fr:inv/102576891) with sleep 21500
216 (oai:icatplus.esrf.fr:inv/102584455) with sleep 21600
217 (oai:icatplus.esrf.fr:inv/103115350) with sleep 21700
218 (oai:icatplus.esrf.fr:inv/103116186) with sleep 21800
219 (oai:icatplus.esrf.fr:inv/103183778) with sleep 21900
220 (oai:icatplus.esrf.fr:inv/103333830) with sleep 22000
221 (oai:icatplus.esrf.fr:inv/103510614) with sleep 22100
222 (oai:icatplus.esrf.fr:inv/103510629) with sleep 22200
223 (oai:icatplus.esrf.fr:inv/103670673) with sleep 22300
224 (oai:icatplus.esrf.fr:inv/103822339) with sleep 22400
225 (oai:icatplus.esrf.fr:inv/103850884) with sleep 22500
226 (oai:icatplus.esrf.fr:inv/104404318) with sleep 22600
227 (oai:icatplus.esrf.fr:inv/104410395) with sleep 22700
228 (oai:icatplus.esrf.fr:inv/104655614) with sleep 22800
229 (oai:icatplus.esrf.fr:inv/104774431) with sleep 22900
230 (oai:icatplus.esrf.fr:inv/104970528) with sleep 23000
231 (oai:icatplus.esrf.fr:inv/104972481) with sleep 23100
232 (oai:icatplus.esrf.fr:inv/105142453) with sleep 23200
233 (oai:icatplus.esrf.fr:inv/105261774) with sleep 23300
234 (oai:icatplus.esrf.fr:inv/105362144) with sleep 23400
235 (oai:icatplus.esrf.fr:inv/105493981) with sleep 23500
236 (oai:icatplus.esrf.fr:inv/105564349) with sleep 23600
237 (oai:icatplus.esrf.fr:inv/105709996) with sleep 23700
238 (oai:icatplus.esrf.fr:inv/105710016) with sleep 23800
239 (oai:icatplus.esrf.fr:inv/105804379) with sleep 23900
240 (oai:icatplus.esrf.fr:inv/105805374) with sleep 24000
241 (oai:icatplus.esrf.fr:inv/106105591) with sleep 24100
242 (oai:icatplus.esrf.fr:inv/106107338) with sleep 24200
243 (oai:icatplus.esrf.fr:inv/106333436) with sleep 24300
244 (oai:icatplus.esrf.fr:inv/107950358) with sleep 24400
245 (oai:icatplus.esrf.fr:inv/108012210) with sleep 24500
246 (oai:icatplus.esrf.fr:inv/108012307) with sleep 24600
247 (oai:icatplus.esrf.fr:inv/108154290) with sleep 24700
248 (oai:icatplus.esrf.fr:inv/108294796) with sleep 24800
249 (oai:icatplus.esrf.fr:inv/108404596) with sleep 24900
250 (oai:icatplus.esrf.fr:inv/108715776) with sleep 25000
251 (oai:icatplus.esrf.fr:inv/108718973) with sleep 25100
252 (oai:icatplus.esrf.fr:inv/108737545) with sleep 25200
253 (oai:icatplus.esrf.fr:inv/108936441) with sleep 25300
254 (oai:icatplus.esrf.fr:inv/109206218) with sleep 25400
255 (oai:icatplus.esrf.fr:inv/109431849) with sleep 25500
256 (oai:icatplus.esrf.fr:inv/109967489) with sleep 25600
257 (oai:icatplus.esrf.fr:inv/110423041) with sleep 25700
258 (oai:icatplus.esrf.fr:inv/110859523) with sleep 25800
259 (oai:icatplus.esrf.fr:inv/111133234) with sleep 25900
260 (oai:icatplus.esrf.fr:inv/111168787) with sleep 26000
261 (oai:icatplus.esrf.fr:inv/111578542) with sleep 26100
262 (oai:icatplus.esrf.fr:inv/111584835) with sleep 26200
263 (oai:icatplus.esrf.fr:inv/112053688) with sleep 26300
264 (oai:icatplus.esrf.fr:inv/112482147) with sleep 26400
265 (oai:icatplus.esrf.fr:inv/112668423) with sleep 26500
266 (oai:icatplus.esrf.fr:inv/112668425) with sleep 26600
267 (oai:icatplus.esrf.fr:inv/112688911) with sleep 26700
268 (oai:icatplus.esrf.fr:inv/113585769) with sleep 26800
269 (oai:icatplus.esrf.fr:inv/113895337) with sleep 26900
270 (oai:icatplus.esrf.fr:inv/113999726) with sleep 27000
271 (oai:icatplus.esrf.fr:inv/114199993) with sleep 27100
272 (oai:icatplus.esrf.fr:inv/114565706) with sleep 27200
273 (oai:icatplus.esrf.fr:inv/114573812) with sleep 27300
274 (oai:icatplus.esrf.fr:inv/114980029) with sleep 27400
275 (oai:icatplus.esrf.fr:inv/115305167) with sleep 27500
276 (oai:icatplus.esrf.fr:inv/115334770) with sleep 27600
277 (oai:icatplus.esrf.fr:inv/115354787) with sleep 27700
278 (oai:icatplus.esrf.fr:inv/115451959) with sleep 27800
279 (oai:icatplus.esrf.fr:inv/115539115) with sleep 27900
280 (oai:icatplus.esrf.fr:inv/115548504) with sleep 28000
281 (oai:icatplus.esrf.fr:inv/115658541) with sleep 28100
282 (oai:icatplus.esrf.fr:inv/115705525) with sleep 28200
283 (oai:icatplus.esrf.fr:inv/116040417) with sleep 28300
284 (oai:icatplus.esrf.fr:inv/116065101) with sleep 28400
285 (oai:icatplus.esrf.fr:inv/116519894) with sleep 28500
286 (oai:icatplus.esrf.fr:inv/116530300) with sleep 28600
287 (oai:icatplus.esrf.fr:inv/116548212) with sleep 28700
288 (oai:icatplus.esrf.fr:inv/116934142) with sleep 28800
289 (oai:icatplus.esrf.fr:inv/116937630) with sleep 28900
290 (oai:icatplus.esrf.fr:inv/116939075) with sleep 29000
291 (oai:icatplus.esrf.fr:inv/117235103) with sleep 29100
292 (oai:icatplus.esrf.fr:inv/117364477) with sleep 29200
293 (oai:icatplus.esrf.fr:inv/117583838) with sleep 29300
294 (oai:icatplus.esrf.fr:inv/117586989) with sleep 29400
295 (oai:icatplus.esrf.fr:inv/117605227) with sleep 29500
296 (oai:icatplus.esrf.fr:inv/118167609) with sleep 29600
297 (oai:icatplus.esrf.fr:inv/118169867) with sleep 29700
298 (oai:icatplus.esrf.fr:inv/118169876) with sleep 29800
299 (oai:icatplus.esrf.fr:inv/118169885) with sleep 29900
300 (oai:icatplus.esrf.fr:inv/118169891) with sleep 30000
301 (oai:icatplus.esrf.fr:inv/118169900) with sleep 30100
302 (oai:icatplus.esrf.fr:inv/118169907) with sleep 30200
303 (oai:icatplus.esrf.fr:inv/118169916) with sleep 30300
304 (oai:icatplus.esrf.fr:inv/118169925) with sleep 30400
305 (oai:icatplus.esrf.fr:inv/118169928) with sleep 30500
306 (oai:icatplus.esrf.fr:inv/118169937) with sleep 30600
307 (oai:icatplus.esrf.fr:inv/118169949) with sleep 30700
308 (oai:icatplus.esrf.fr:inv/118169953) with sleep 30800
309 (oai:icatplus.esrf.fr:inv/118169962) with sleep 30900
310 (oai:icatplus.esrf.fr:inv/118169965) with sleep 31000
311 (oai:icatplus.esrf.fr:inv/118169971) with sleep 31100
312 (oai:icatplus.esrf.fr:inv/118169974) with sleep 31200
313 (oai:icatplus.esrf.fr:inv/118169986) with sleep 31300
314 (oai:icatplus.esrf.fr:inv/118169992) with sleep 31400
315 (oai:icatplus.esrf.fr:inv/118169999) with sleep 31500
316 (oai:icatplus.esrf.fr:inv/118170174) with sleep 31600
317 (oai:icatplus.esrf.fr:inv/118179781) with sleep 31700
318 (oai:icatplus.esrf.fr:inv/118346851) with sleep 31800
319 (oai:icatplus.esrf.fr:inv/118368616) with sleep 31900
320 (oai:icatplus.esrf.fr:inv/118447013) with sleep 32000
321 (oai:icatplus.esrf.fr:inv/118447194) with sleep 32100
322 (oai:icatplus.esrf.fr:inv/118498674) with sleep 32200
323 (oai:icatplus.esrf.fr:inv/118629616) with sleep 32300
324 (oai:icatplus.esrf.fr:inv/118686985) with sleep 32400
325 (oai:icatplus.esrf.fr:inv/118729694) with sleep 32500
326 (oai:icatplus.esrf.fr:inv/118923391) with sleep 32600
327 (oai:icatplus.esrf.fr:inv/118934380) with sleep 32700
328 (oai:icatplus.esrf.fr:inv/119235097) with sleep 32800
329 (oai:icatplus.esrf.fr:inv/119463983) with sleep 32900
330 (oai:icatplus.esrf.fr:inv/119464351) with sleep 33000
331 (oai:icatplus.esrf.fr:inv/119464359) with sleep 33100
332 (oai:icatplus.esrf.fr:inv/119499096) with sleep 33200
333 (oai:icatplus.esrf.fr:inv/119568907) with sleep 33300
334 (oai:icatplus.esrf.fr:inv/119804980) with sleep 33400
335 (oai:icatplus.esrf.fr:inv/119805491) with sleep 33500
336 (oai:icatplus.esrf.fr:inv/119805810) with sleep 33600
337 (oai:icatplus.esrf.fr:inv/119895319) with sleep 33700
338 (oai:icatplus.esrf.fr:inv/119994641) with sleep 33800
339 (oai:icatplus.esrf.fr:inv/120104269) with sleep 33900
340 (oai:icatplus.esrf.fr:inv/120120588) with sleep 34000
341 (oai:icatplus.esrf.fr:inv/120126396) with sleep 34100
342 (oai:icatplus.esrf.fr:inv/120403650) with sleep 34200
343 (oai:icatplus.esrf.fr:inv/120878197) with sleep 34300
344 (oai:icatplus.esrf.fr:inv/120961707) with sleep 34400
345 (oai:icatplus.esrf.fr:inv/121587363) with sleep 34500
346 (oai:icatplus.esrf.fr:inv/121797240) with sleep 34600
347 (oai:icatplus.esrf.fr:inv/121810241) with sleep 34700
348 (oai:icatplus.esrf.fr:inv/122135087) with sleep 34800
349 (oai:icatplus.esrf.fr:inv/122136876) with sleep 34900
350 (oai:icatplus.esrf.fr:inv/122238319) with sleep 35000
351 (oai:icatplus.esrf.fr:inv/122471822) with sleep 35100
352 (oai:icatplus.esrf.fr:inv/122481661) with sleep 35200
353 (oai:icatplus.esrf.fr:inv/122496872) with sleep 35300
354 (oai:icatplus.esrf.fr:inv/123016643) with sleep 35400
355 (oai:icatplus.esrf.fr:inv/124222055) with sleep 35500
356 (oai:icatplus.esrf.fr:inv/124227492) with sleep 35600
357 (oai:icatplus.esrf.fr:inv/124265259) with sleep 35700
358 (oai:icatplus.esrf.fr:inv/124358337) with sleep 35800
359 (oai:icatplus.esrf.fr:inv/126193516) with sleep 35900
360 (oai:icatplus.esrf.fr:inv/126600748) with sleep 36000
361 (oai:icatplus.esrf.fr:inv/126693670) with sleep 36100
362 (oai:icatplus.esrf.fr:inv/126791309) with sleep 36200
363 (oai:icatplus.esrf.fr:inv/126868849) with sleep 36300
364 (oai:icatplus.esrf.fr:inv/126885646) with sleep 36400
365 (oai:icatplus.esrf.fr:inv/126886707) with sleep 36500
366 (oai:icatplus.esrf.fr:inv/127019713) with sleep 36600
367 (oai:icatplus.esrf.fr:inv/127315716) with sleep 36700
368 (oai:icatplus.esrf.fr:inv/127319578) with sleep 36800
369 (oai:icatplus.esrf.fr:inv/127328736) with sleep 36900
370 (oai:icatplus.esrf.fr:inv/127495001) with sleep 37000
371 (oai:icatplus.esrf.fr:inv/127754215) with sleep 37100
372 (oai:icatplus.esrf.fr:inv/127835087) with sleep 37200
373 (oai:icatplus.esrf.fr:inv/127992737) with sleep 37300
374 (oai:icatplus.esrf.fr:inv/127993035) with sleep 37400
375 (oai:icatplus.esrf.fr:inv/127993606) with sleep 37500
376 (oai:icatplus.esrf.fr:inv/127995557) with sleep 37600
377 (oai:icatplus.esrf.fr:inv/128500156) with sleep 37700
378 (oai:icatplus.esrf.fr:inv/128504822) with sleep 37800
379 (oai:icatplus.esrf.fr:inv/128944986) with sleep 37900
380 (oai:icatplus.esrf.fr:inv/128998416) with sleep 38000
381 (oai:icatplus.esrf.fr:inv/129117363) with sleep 38100
382 (oai:icatplus.esrf.fr:inv/129233760) with sleep 38200
383 (oai:icatplus.esrf.fr:inv/130026137) with sleep 38300
384 (oai:icatplus.esrf.fr:inv/130033368) with sleep 38400
385 (oai:icatplus.esrf.fr:inv/130412444) with sleep 38500
386 (oai:icatplus.esrf.fr:inv/130415014) with sleep 38600
387 (oai:icatplus.esrf.fr:inv/130838861) with sleep 38700
388 (oai:icatplus.esrf.fr:inv/130874477) with sleep 38800
389 (oai:icatplus.esrf.fr:inv/130874981) with sleep 38900
390 (oai:icatplus.esrf.fr:inv/130874985) with sleep 39000
391 (oai:icatplus.esrf.fr:inv/130884741) with sleep 39100
392 (oai:icatplus.esrf.fr:inv/130997639) with sleep 39200
393 (oai:icatplus.esrf.fr:inv/131009905) with sleep 39300
394 (oai:icatplus.esrf.fr:inv/131022975) with sleep 39400
395 (oai:icatplus.esrf.fr:inv/131077401) with sleep 39500
396 (oai:icatplus.esrf.fr:inv/131150513) with sleep 39600
397 (oai:icatplus.esrf.fr:inv/131846968) with sleep 39700
398 (oai:icatplus.esrf.fr:inv/132183231) with sleep 39800
399 (oai:icatplus.esrf.fr:inv/132279991) with sleep 39900
400 (oai:icatplus.esrf.fr:inv/132283130) with sleep 40000
401 (oai:icatplus.esrf.fr:inv/132528576) with sleep 40100
402 (oai:icatplus.esrf.fr:inv/132528770) with sleep 40200
403 (oai:icatplus.esrf.fr:inv/132630521) with sleep 40300
404 (oai:icatplus.esrf.fr:inv/132862896) with sleep 40400
405 (oai:icatplus.esrf.fr:inv/132939048) with sleep 40500
406 (oai:icatplus.esrf.fr:inv/132948299) with sleep 40600
407 (oai:icatplus.esrf.fr:inv/133044090) with sleep 40700
408 (oai:icatplus.esrf.fr:inv/133065895) with sleep 40800
409 (oai:icatplus.esrf.fr:inv/133417050) with sleep 40900
410 (oai:icatplus.esrf.fr:inv/133417381) with sleep 41000
411 (oai:icatplus.esrf.fr:inv/133418343) with sleep 41100
412 (oai:icatplus.esrf.fr:inv/133418463) with sleep 41200
413 (oai:icatplus.esrf.fr:inv/133424186) with sleep 41300
414 (oai:icatplus.esrf.fr:inv/133436389) with sleep 41400
415 (oai:icatplus.esrf.fr:inv/133628388) with sleep 41500
416 (oai:icatplus.esrf.fr:inv/133989980) with sleep 41600
417 (oai:icatplus.esrf.fr:inv/134461532) with sleep 41700
418 (oai:icatplus.esrf.fr:inv/134500785) with sleep 41800
419 (oai:icatplus.esrf.fr:inv/134751624) with sleep 41900
420 (oai:icatplus.esrf.fr:inv/134958966) with sleep 42000
421 (oai:icatplus.esrf.fr:inv/135135830) with sleep 42100
422 (oai:icatplus.esrf.fr:inv/135588471) with sleep 42200
423 (oai:icatplus.esrf.fr:inv/135588622) with sleep 42300
424 (oai:icatplus.esrf.fr:inv/135760149) with sleep 42400
425 (oai:icatplus.esrf.fr:inv/135806187) with sleep 42500
426 (oai:icatplus.esrf.fr:inv/135806642) with sleep 42600
427 (oai:icatplus.esrf.fr:inv/135816585) with sleep 42700
428 (oai:icatplus.esrf.fr:inv/137205237) with sleep 42800
429 (oai:icatplus.esrf.fr:inv/137293723) with sleep 42900
430 (oai:icatplus.esrf.fr:inv/137317923) with sleep 43000
431 (oai:icatplus.esrf.fr:inv/137827701) with sleep 43100
432 (oai:icatplus.esrf.fr:inv/138040476) with sleep 43200
433 (oai:icatplus.esrf.fr:inv/138779835) with sleep 43300
434 (oai:icatplus.esrf.fr:inv/139035493) with sleep 43400
435 (oai:icatplus.esrf.fr:inv/140296588) with sleep 43500
436 (oai:icatplus.esrf.fr:inv/140360388) with sleep 43600
437 (oai:icatplus.esrf.fr:inv/140436170) with sleep 43700
438 (oai:icatplus.esrf.fr:inv/140506618) with sleep 43800
439 (oai:icatplus.esrf.fr:inv/141963373) with sleep 43900
440 (oai:icatplus.esrf.fr:inv/141972475) with sleep 44000
441 (oai:icatplus.esrf.fr:inv/142481073) with sleep 44100
442 (oai:icatplus.esrf.fr:inv/142673807) with sleep 44200
443 (oai:icatplus.esrf.fr:inv/142757577) with sleep 44300
444 (oai:icatplus.esrf.fr:inv/142840375) with sleep 44400
445 (oai:icatplus.esrf.fr:inv/142840456) with sleep 44500
446 (oai:icatplus.esrf.fr:inv/142846529) with sleep 44600
447 (oai:icatplus.esrf.fr:inv/142866117) with sleep 44700
448 (oai:icatplus.esrf.fr:inv/143094721) with sleep 44800
449 (oai:icatplus.esrf.fr:inv/143360964) with sleep 44900
450 (oai:icatplus.esrf.fr:inv/143373441) with sleep 45000
451 (oai:icatplus.esrf.fr:inv/143436284) with sleep 45100
452 (oai:icatplus.esrf.fr:inv/144367705) with sleep 45200
453 (oai:icatplus.esrf.fr:inv/144479045) with sleep 45300
454 (oai:icatplus.esrf.fr:inv/144554304) with sleep 45400
455 (oai:icatplus.esrf.fr:inv/144580587) with sleep 45500
456 (oai:icatplus.esrf.fr:inv/144796048) with sleep 45600
457 (oai:icatplus.esrf.fr:inv/145085063) with sleep 45700
458 (oai:icatplus.esrf.fr:inv/145085249) with sleep 45800
459 (oai:icatplus.esrf.fr:inv/145088681) with sleep 45900
460 (oai:icatplus.esrf.fr:inv/145107284) with sleep 46000
461 (oai:icatplus.esrf.fr:inv/145194304) with sleep 46100
462 (oai:icatplus.esrf.fr:inv/145267148) with sleep 46200
463 (oai:icatplus.esrf.fr:inv/145450145) with sleep 46300
464 (oai:icatplus.esrf.fr:inv/146277404) with sleep 46400
465 (oai:icatplus.esrf.fr:inv/146282714) with sleep 46500
466 (oai:icatplus.esrf.fr:inv/146284282) with sleep 46600
467 (oai:icatplus.esrf.fr:inv/146304266) with sleep 46700
468 (oai:icatplus.esrf.fr:inv/146788748) with sleep 46800
469 (oai:icatplus.esrf.fr:inv/147110737) with sleep 46900
470 (oai:icatplus.esrf.fr:inv/147110748) with sleep 47000
471 (oai:icatplus.esrf.fr:inv/147122999) with sleep 47100
472 (oai:icatplus.esrf.fr:inv/147124103) with sleep 47200
473 (oai:icatplus.esrf.fr:inv/147900374) with sleep 47300
474 (oai:icatplus.esrf.fr:inv/149283553) with sleep 47400
475 (oai:icatplus.esrf.fr:inv/150269629) with sleep 47500
476 (oai:icatplus.esrf.fr:inv/150291534) with sleep 47600
477 (oai:icatplus.esrf.fr:inv/150325184) with sleep 47700
478 (oai:icatplus.esrf.fr:inv/150860761) with sleep 47800
479 (oai:icatplus.esrf.fr:inv/151542137) with sleep 47900
480 (oai:icatplus.esrf.fr:inv/151565774) with sleep 48000
481 (oai:icatplus.esrf.fr:inv/152611353) with sleep 48100
482 (oai:icatplus.esrf.fr:inv/154128138) with sleep 48200
483 (oai:icatplus.esrf.fr:inv/154202231) with sleep 48300
484 (oai:icatplus.esrf.fr:inv/155204753) with sleep 48400
485 (oai:icatplus.esrf.fr:inv/155205381) with sleep 48500
486 (oai:icatplus.esrf.fr:inv/155205493) with sleep 48600
487 (oai:icatplus.esrf.fr:inv/155216741) with sleep 48700
488 (oai:icatplus.esrf.fr:inv/155240127) with sleep 48800
489 (oai:icatplus.esrf.fr:inv/155242587) with sleep 48900
490 (oai:icatplus.esrf.fr:inv/155404944) with sleep 49000
491 (oai:icatplus.esrf.fr:inv/157299629) with sleep 49100
492 (oai:icatplus.esrf.fr:inv/157689375) with sleep 49200
493 (oai:icatplus.esrf.fr:inv/157726842) with sleep 49300
494 (oai:icatplus.esrf.fr:inv/157726885) with sleep 49400
495 (oai:icatplus.esrf.fr:inv/157904329) with sleep 49500
496 (oai:icatplus.esrf.fr:inv/157970994) with sleep 49600
497 (oai:icatplus.esrf.fr:inv/158028007) with sleep 49700
498 (oai:icatplus.esrf.fr:inv/158308267) with sleep 49800
499 (oai:icatplus.esrf.fr:inv/158573495) with sleep 49900
500 (oai:icatplus.esrf.fr:inv/158859089) with sleep 50000

Code:

const oaipmhResource = require("../resources/oaipmh.resource.js");
const sessionHelper = require('../helper/session.js');
const XmlReader = require('xml-reader');
const XmlQuery = require('xml-query');
require('it-each')({
    testPerIteration: true
});

describe('OAI-PMH', () => {
    describe('/oai-pmh request', () => {
        it.each(oaipmhResource.requests, '[%s]', ['description'], function (element, next) {
       // Request the list of identifiers
            global.gRequester
            .get('/oaipmh/'+ element.requestListIdentifiers)
            .send()
            .end((err, getOAIResponse) => {     
        expect(getOAIResponse.status).to.equal(200);                            
        let identifiers = (XmlQuery(XmlReader.parseSync(getOAIResponse.text)).find('identifier'));
        let counter = 1;
                let resolved = 0;
        console.log("#identifiers:" + identifiers.ast.length)
        identifiers.each(identifier => {
            // Request by investigation identifier
            var doRequest = function(id, t_sleep, c){
                 global.gRequester
                        .get('/oaipmh/'+ element.requestByIdentifier + id)
                        .send()
                        .end((err, getInvestigationResponse) => {
                        expect(getInvestigationResponse.status).to.equal(200);    
                        resolved = resolved + 1;
                        console.log(resolved + " (" + id +  ") with sleep " + t_sleep)
                        let title = (XmlQuery(XmlReader.parseSync(getInvestigationResponse.text)).find('dc:title').text());
                        // check that there are no forbidden proposal types
                        element.expected.forbiddenProposalTypes.forEach(forbiddenProposalType => {
                            expect(title.toLowerCase().startsWith(forbiddenProposalType)).to.be.false;
                        });
                        // end of tests condition
                        if (counter == identifiers.ast.length){
                            next();
                        }

                })
            }
            let sleep = counter * 100;
            counter = counter + 1;
            setTimeout(function(){doRequest(identifier.children[0].value, sleep, JSON.stringify(counter));}, sleep)

        });              

            });

        }
    )

 });
RKrahl commented 4 years ago

Just to check that I understand that right: you were sending the 500 GetRecord requests in parallel?

antolinos commented 4 years ago

Correct. This is what I imagine harvesters will do, right? If we expose millions of records....

RKrahl commented 4 years ago

No, I would not expect any harvester to do that. Why should they?

antolinos commented 4 years ago

I don't know, how are they supposed to harvest the data?

RKrahl commented 4 years ago

First of all, if they want all the records, I would expect them to do one single ListRecords call rather then a ListIdentifiers plus individual GetRecord calls. Fetching all the records is exactly what ListRecords does. Secondly, even if they would do individual GetRecords calls, it simply makes no sense to do more then 10 or so in parallel, because you wouldn't get more of them through the wire simultaneously anyway.

antolinos commented 4 years ago

Ok. I understand. I will add ListRecords to my tests. Should I test anything else?