search

bmuller / twistar

Twistar is an object-relational mapper (ORM) for Python that uses the Twisted library to provide asynchronous DB interaction.
http://findingscience.com/twistar
Other
132 stars 38 forks source link

validatesUniquenessOf triggers Segmentation Fault in a particular case #66

Closed Reve closed 8 years ago

Reve commented 8 years ago

The seg fault is triggered when you try to validate the uniqueness of a random string ("xn--tan-test-20160307154100-6tc060eub.ro"). validatePresenceOf works well with the same string provided. By looking at the stack trace, I think the problem is related to the Inflector but I am not 100% sure.

I will attach the stack trace below:

Thank you

`(gdb) bt

0 call_function (f=, throwflag=) at Python/ceval.c:3796

1 PyEval_EvalFrameEx (f=, throwflag=) at Python/ceval.c:2453

2 0x0000003cda2d7647 in PyEval_EvalCodeEx (co=0x1d18288, globals=, locals=, args=, argcount=1,

kws=0x1ddfdf0, kwcount=0, defs=0x0, defcount=0, closure=0x0) at Python/ceval.c:3044

3 0x0000003cda26ad9d in function_call (func=<function at remote 0x1d25848>, arg=

(<Domain(_config=<PostgreSQLDBConfig(txn=None) at remote 0x1d78ea8>, errors=<Errors(infl=<Inflector(Inflector=<English() at remote 0x1d6ff38>) at remote 0x1d6ffc8>) at remote 0x1ddfce0>, user_id=277, fqdn='xn--tan-test-20160307154100-6tc060eub.ro', _deleted=False, id=None) at remote 0x1d70b50>,), kw={})
at Objects/funcobject.c:524

4 0x0000003cda243c63 in PyObject_Call (func=<function at remote 0x1d25848>, arg=, kw=) at Objects/abstract.c:2492

5 0x0000003cda2d4460 in ext_do_call (f=, throwflag=) at Python/ceval.c:4107

6 PyEval_EvalFrameEx (f=, throwflag=) at Python/ceval.c:2493

7 0x0000003cda2d7647 in PyEval_EvalCodeEx (co=0x7f01f06e7378, globals=, locals=, args=,

argcount=1, kws=0x1e09cc8, kwcount=0, defs=0x0, defcount=0, closure=0x0) at Python/ceval.c:3044

8 0x0000003cda2d5a94 in fast_function (f=, throwflag=) at Python/ceval.c:3890

9 call_function (f=, throwflag=) at Python/ceval.c:3815

10 PyEval_EvalFrameEx (f=, throwflag=) at Python/ceval.c:2453

11 0x0000003cda2d7647 in PyEval_EvalCodeEx (co=0x1d0d288, globals=, locals=, args=, argcount=1,

kws=0x1e09ae0, kwcount=0, defs=0x0, defcount=0, closure=0x0) at Python/ceval.c:3044

12 0x0000003cda2d5a94 in fast_function (f=, throwflag=) at Python/ceval.c:3890

13 call_function (f=, throwflag=) at Python/ceval.c:3815

14 PyEval_EvalFrameEx (f=, throwflag=) at Python/ceval.c:2453

15 0x0000003cda2d7647 in PyEval_EvalCodeEx (co=0x1d05c60, globals=, locals=, args=, argcount=1,

kws=0x17e4740, kwcount=0, defs=0x0, defcount=0, closure=(<cell at remote 0x1d7b4e8>,)) at Python/ceval.c:3044

16 0x0000003cda26ad9d in function_call (func=<function at remote 0x1d73320>, arg=(True,), kw={}) at Objects/funcobject.c:524

17 0x0000003cda243c63 in PyObject_Call (func=<function at remote 0x1d73320>, arg=, kw=) at Objects/abstract.c:2492

18 0x0000003cda2d4460 in ext_do_call (f=, throwflag=) at Python/ceval.c:4107

19 PyEval_EvalFrameEx (f=, throwflag=) at Python/ceval.c:2493

20 0x0000003cda2d7647 in PyEval_EvalCodeEx (co=0x7f01f06e7d50, globals=, locals=, args=,

argcount=1, kws=0x1e08ab8, kwcount=0, defs=0x0, defcount=0, closure=0x0) at Python/ceval.c:3044

21 0x0000003cda2d5a94 in fast_function (f=, throwflag=) at Python/ceval.c:3890

22 call_function (f=, throwflag=) at Python/ceval.c:3815

23 PyEval_EvalFrameEx (f=, throwflag=) at Python/ceval.c:2453

24 0x0000003cda2d7647 in PyEval_EvalCodeEx (co=0x7f01f06e7c60, globals=, locals=, args=,

argcount=2, kws=0x1e088e8, kwcount=0, defs=0x0, defcount=0, closure=0x0) at Python/ceval.c:3044

25 0x0000003cda2d5a94 in fast_function (f=, throwflag=) at Python/ceval.c:3890

26 call_function (f=, throwflag=) at Python/ceval.c:3815

27 PyEval_EvalFrameEx (f=, throwflag=) at Python/ceval.c:2453

28 0x0000003cda2d7647 in PyEval_EvalCodeEx (co=0x7f01f06e7918, globals=, locals=, args=,

argcount=2, kws=0x1e08708, kwcount=0, defs=0x0, defcount=0, closure=0x0) at Python/ceval.c:3044

29 0x0000003cda2d5a94 in fast_function (f=, throwflag=) at Python/ceval.c:3890

30 call_function (f=, throwflag=) at Python/ceval.c:3815

31 PyEval_EvalFrameEx (f=, throwflag=) at Python/ceval.c:2453

32 0x0000003cda2d7647 in PyEval_EvalCodeEx (co=0x7f01f02583f0, globals=, locals=, args=,

argcount=4, kws=0x11d9880, kwcount=0, defs=0x0, defcount=0, closure=0x0) at Python/ceval.c:3044

33 0x0000003cda26ad9d in function_call (func=<function at remote 0x7f01f0263410>, arg=

(<DeferredList(finishedCount=2, _chainedTo=None, fireOnOneCallback=False, resultList=[(True, None), (True, None)], _runningCallbacks=True, _canceller=None, callbacks=[((<function at remote 0x1d6ced8>, (), {}), (<function at remote 0x1d6ced8>, (...), {...}))], fireOnOneErrback=False, result=True, _deferredList=[<Deferred(_chainedTo=None, called=True, _canceller=None, callbacks=[], result=None, _runningCallbacks=False) at remote 0x1d6fef0>, <DeferredList(finishedCount=1, _chainedTo=None, fireOnOneCallback=False, resultList=[(True, None)], _runningCallbacks=True, _canceller=None, callbacks=[], fireOnOneErrback=False, result=None, _deferredList=[<Deferred(_chainedTo=None, called=True, _canceller=None, callbacks=[], result=None, _runningCallbacks=True) at remote 0x1d6fea8>], called=True, consumeErrors=False) at remote 0x1d780e0>], called=True, consumeErrors=False) at remote 0x1d78e60>, None, 1, True), kw={}) at Objects/funcobject.c:524

34 0x0000003cda243c63 in PyObject_Call (func=<function at remote 0x7f01f0263410>, arg=, kw=) at Objects/abstract.c:2492

---Type to continue, or q to quit---

35 0x0000003cda2d4460 in ext_do_call (f=, throwflag=) at Python/ceval.c:4107

36 PyEval_EvalFrameEx (f=, throwflag=) at Python/ceval.c:2493

37 0x0000003cda2d7647 in PyEval_EvalCodeEx (co=0x7f01f06e7d50, globals=, locals=, args=,

argcount=1, kws=0x1dcf7b8, kwcount=0, defs=0x0, defcount=0, closure=0x0) at Python/ceval.c:3044

38 0x0000003cda2d5a94 in fast_function (f=, throwflag=) at Python/ceval.c:3890

39 call_function (f=, throwflag=) at Python/ceval.c:3815

40 PyEval_EvalFrameEx (f=, throwflag=) at Python/ceval.c:2453

41 0x0000003cda2d7647 in PyEval_EvalCodeEx (co=0x7f01f06e7c60, globals=, locals=, args=,

argcount=2, kws=0x1e076b8, kwcount=0, defs=0x0, defcount=0, closure=0x0) at Python/ceval.c:3044

42 0x0000003cda2d5a94 in fast_function (f=, throwflag=) at Python/ceval.c:3890

43 call_function (f=, throwflag=) at Python/ceval.c:3815

44 PyEval_EvalFrameEx (f=, throwflag=) at Python/ceval.c:2453

45 0x0000003cda2d7647 in PyEval_EvalCodeEx (co=0x7f01f06e7918, globals=, locals=, args=,

argcount=2, kws=0x1de5008, kwcount=0, defs=0x0, defcount=0, closure=0x0) at Python/ceval.c:3044

46 0x0000003cda2d5a94 in fast_function (f=, throwflag=) at Python/ceval.c:3890

47 call_function (f=, throwflag=) at Python/ceval.c:3815

48 PyEval_EvalFrameEx (f=, throwflag=) at Python/ceval.c:2453

49 0x0000003cda2d7647 in PyEval_EvalCodeEx (co=0x7f01f02583f0, globals=, locals=, args=,

argcount=4, kws=0x1d4d8b0, kwcount=0, defs=0x0, defcount=0, closure=0x0) at Python/ceval.c:3044

50 0x0000003cda26ad9d in function_call (func=<function at remote 0x7f01f0263410>, arg=

(<DeferredList(finishedCount=1, _chainedTo=None, fireOnOneCallback=False, resultList=[(True, None)], _runningCallbacks=True, _canceller=None, callbacks=[], fireOnOneErrback=False, result=None, _deferredList=[<Deferred(_chainedTo=None, called=True, _canceller=None, callbacks=[], result=None, _runningCallbacks=True) at remote 0x1d6fea8>], called=True, consumeErrors=False) at remote 0x1d780e0>, None, 0, True), kw={}) at Objects/funcobject.c:524

51 0x0000003cda243c63 in PyObject_Call (func=<function at remote 0x7f01f0263410>, arg=, kw=) at Objects/abstract.c:2492

52 0x0000003cda2d4460 in ext_do_call (f=, throwflag=) at Python/ceval.c:4107

53 PyEval_EvalFrameEx (f=, throwflag=) at Python/ceval.c:2493

54 0x0000003cda2d7647 in PyEval_EvalCodeEx (co=0x7f01f06e7d50, globals=, locals=, args=,

argcount=1, kws=0x1de4bc8, kwcount=0, defs=0x0, defcount=0, closure=0x0) at Python/ceval.c:3044

55 0x0000003cda2d5a94 in fast_function (f=, throwflag=) at Python/ceval.c:3890

56 call_function (f=, throwflag=) at Python/ceval.c:3815

57 PyEval_EvalFrameEx (f=, throwflag=) at Python/ceval.c:2453

58 0x0000003cda2d7647 in PyEval_EvalCodeEx (co=0x7f01f06e7c60, globals=, locals=, args=,

argcount=2, kws=0x1de44b8, kwcount=0, defs=0x0, defcount=0, closure=0x0) at Python/ceval.c:3044

59 0x0000003cda2d5a94 in fast_function (f=, throwflag=) at Python/ceval.c:3890

60 call_function (f=, throwflag=) at Python/ceval.c:3815

61 PyEval_EvalFrameEx (f=, throwflag=) at Python/ceval.c:2453

62 0x0000003cda2d7647 in PyEval_EvalCodeEx (co=0x7f01f06e7918, globals=, locals=, args=,

argcount=2, kws=0x1dbab50, kwcount=0, defs=0x0, defcount=0, closure=0x0) at Python/ceval.c:3044

63 0x0000003cda26ad9d in function_call (func=<function at remote 0x7f01f0264c08>, arg=

(<Deferred(_chainedTo=None, called=True, _canceller=None, callbacks=[], result=None, _runningCallbacks=True) at remote 0x1d6fea8>, None), kw={})
at Objects/funcobject.c:524

64 0x0000003cda243c63 in PyObject_Call (func=<function at remote 0x7f01f0264c08>, arg=, kw=) at Objects/abstract.c:2492

65 0x0000003cda2d4460 in ext_do_call (f=, throwflag=) at Python/ceval.c:4107

66 PyEval_EvalFrameEx (f=, throwflag=) at Python/ceval.c:2493

67 0x0000003cda2d7647 in PyEval_EvalCodeEx (co=0x16db7b0, globals=, locals=, args=, argcount=1,

kws=0x1dcf4a8, kwcount=0, defs=0x0, defcount=0, closure=0x0) at Python/ceval.c:3044

68 0x0000003cda2d5a94 in fast_function (f=, throwflag=) at Python/ceval.c:3890

69 call_function (f=, throwflag=) at Python/ceval.c:3815

70 PyEval_EvalFrameEx (f=, throwflag=) at Python/ceval.c:2453

71 0x0000003cda2d7647 in PyEval_EvalCodeEx (co=0x16e0a80, globals=, locals=, args=, argcount=1,

kws=0x1dcde80, kwcount=0, defs=0x0, defcount=0, closure=0x0) at Python/ceval.c:3044

72 0x0000003cda2d5a94 in fast_function (f=, throwflag=) at Python/ceval.c:3890

---Type to continue, or q to quit---

73 call_function (f=, throwflag=) at Python/ceval.c:3815

74 PyEval_EvalFrameEx (f=, throwflag=) at Python/ceval.c:2453

75 0x0000003cda2d7647 in PyEval_EvalCodeEx (co=0x16e0990, globals=, locals=, args=, argcount=1,

kws=0x1dcdc90, kwcount=0, defs=0x16f7728, defcount=1, closure=0x0) at Python/ceval.c:3044

76 0x0000003cda2d5a94 in fast_function (f=, throwflag=) at Python/ceval.c:3890

77 call_function (f=, throwflag=) at Python/ceval.c:3815

78 PyEval_EvalFrameEx (f=, throwflag=) at Python/ceval.c:2453

79 0x0000003cda2d7647 in PyEval_EvalCodeEx (co=0x7f01f273fcd8, globals=, locals=, args=,

argcount=5, kws=0x1dcdaa8, kwcount=0, defs=0x7f01f06ce188, defcount=2, closure=0x0) at Python/ceval.c:3044

80 0x0000003cda2d5a94 in fast_function (f=, throwflag=) at Python/ceval.c:3890

81 call_function (f=, throwflag=) at Python/ceval.c:3815

82 PyEval_EvalFrameEx (f=, throwflag=) at Python/ceval.c:2453

83 0x0000003cda2d7647 in PyEval_EvalCodeEx (co=0x7f01f2745030, globals=, locals=, args=,

argcount=4, kws=0x1dbe958, kwcount=0, defs=0x0, defcount=0, closure=0x0) at Python/ceval.c:3044

84 0x0000003cda2d5a94 in fast_function (f=, throwflag=) at Python/ceval.c:3890

85 call_function (f=, throwflag=) at Python/ceval.c:3815

86 PyEval_EvalFrameEx (f=, throwflag=) at Python/ceval.c:2453

87 0x0000003cda2d6b7f in fast_function (f=, throwflag=) at Python/ceval.c:3880

88 call_function (f=, throwflag=) at Python/ceval.c:3815

89 PyEval_EvalFrameEx (f=, throwflag=) at Python/ceval.c:2453

90 0x0000003cda2d7647 in PyEval_EvalCodeEx (co=0x7f01f273ff30, globals=, locals=, args=,

argcount=1, kws=0x140f258, kwcount=0, defs=0x0, defcount=0, closure=0x0) at Python/ceval.c:3044

91 0x0000003cda2d5a94 in fast_function (f=, throwflag=) at Python/ceval.c:3890

92 call_function (f=, throwflag=) at Python/ceval.c:3815

93 PyEval_EvalFrameEx (f=, throwflag=) at Python/ceval.c:2453

94 0x0000003cda2d6b7f in fast_function (f=, throwflag=) at Python/ceval.c:3880

95 call_function (f=, throwflag=) at Python/ceval.c:3815

96 PyEval_EvalFrameEx (f=, throwflag=) at Python/ceval.c:2453

97 0x0000003cda2d7647 in PyEval_EvalCodeEx (co=0x7f01f2745a08, globals=, locals=, args=,

argcount=2, kws=0x1416a60, kwcount=0, defs=0x0, defcount=0, closure=0x0) at Python/ceval.c:3044

98 0x0000003cda2d5a94 in fast_function (f=, throwflag=) at Python/ceval.c:3890

99 call_function (f=, throwflag=) at Python/ceval.c:3815

100 PyEval_EvalFrameEx (f=, throwflag=) at Python/ceval.c:2453

101 0x0000003cda2d6b7f in fast_function (f=, throwflag=) at Python/ceval.c:3880

102 call_function (f=, throwflag=) at Python/ceval.c:3815

103 PyEval_EvalFrameEx (f=, throwflag=) at Python/ceval.c:2453

104 0x0000003cda2d7647 in PyEval_EvalCodeEx (co=0x7f01f3e8a558, globals=, locals=, args=,

argcount=0, kws=0x0, kwcount=0, defs=0x0, defcount=0, closure=0x0) at Python/ceval.c:3044

105 0x0000003cda2d7722 in PyEval_EvalCode (co=, globals=, locals=) at Python/ceval.c:545

106 0x0000003cda2f1b9c in run_mod (mod=, filename=, globals=

{'run': <function at remote 0x7f01f2745230>, '__builtins__': <module at remote 0x7f01f3f3c868>, '__file__': '/usr/bin/twistd', '__package__': None, 'sys': <module at remote 0x7f01f3f3cc20>, '__name__': '__main__', 'os': <module at remote 0x7f01f3eef9b8>, '__doc__': None}, locals=
{'run': <function at remote 0x7f01f2745230>, '__builtins__': <module at remote 0x7f01f3f3c868>, '__file__': '/usr/bin/twistd', '__package__': None, 'sys': <module at remote 0x7f01f3f3cc20>, '__name__': '__main__', 'os': <module at remote 0x7f01f3eef9b8>, '__doc__': None}, flags=<value optimized out>,
arena=<value optimized out>) at Python/pythonrun.c:1358

107 0x0000003cda2f1c70 in PyRun_FileExFlags (fp=0x1097630, filename=0x7ffd72c495f3 "/usr/bin/twistd", start=, globals=

{'run': <function at remote 0x7f01f2745230>, '__builtins__': <module at remote 0x7f01f3f3c868>, '__file__': '/usr/bin/twistd', '__package__': None, 'sys': <module at remote 0x7f01f3f3cc20>, '__name__': '__main__', 'os': <module at remote 0x7f01f3eef9b8>, '__doc__': None}, locals=
{'run': <function at remote 0x7f01f2745230>, '__builtins__': <module at remote 0x7f01f3f3c868>, '__file__': '/usr/bin/twistd', '__package__': None, 'sys': <module at remote 0x7f01f3f3cc20>, '__name__': '__main__', 'os': <module at remote 0x7f01f3eef9b8>, '__doc__': None}, closeit=1, flags=0x7ffd72c48520)
at Python/pythonrun.c:1344

108 0x0000003cda2f315c in PyRun_SimpleFileExFlags (fp=0x1097630, filename=0x7ffd72c495f3 "/usr/bin/twistd", closeit=1, flags=0x7ffd72c48520)

---Type to continue, or q to quit--- at Python/pythonrun.c:948

109 0x0000003cda2ff892 in Py_Main (argc=, argv=) at Modules/main.c:618

110 0x00000039cd21ed5d in __libc_start_main (main=0x400710
, argc=6, ubp_av=0x7ffd72c48648, init=, fini=,

rtld_fini=<value optimized out>, stack_end=0x7ffd72c48638) at libc-start.c:226

111 0x0000000000400649 in _start ()`

bmuller commented 8 years ago

Thanks @Reve - do you have an example script that will demonstrate this?

Reve commented 8 years ago 
from twistar.dbobject import DBObject

class Domain(DBObject):
    TABLENAME = 'domain'
    HABTM = ['tags']

    searchable_fields = ['id', 'user_id', 'fqdn', 'crdate']
    sortable_fields = ['user_id', 'fqdn', 'crdate']

    id = None
    user_id = None
    fqdn = None
    crdate = None

Domain.validatesPresenceOf('user_id', 'fqdn')
# This here throws segmentation fault
Domain.validatesUniquenessOf('fqdn')

And this is called in a command. The script dies on save().

db_domain = Domain()
db_domain.user_id = _user_id
db_domain.fqdn = domain
dom = yield db_domain.save()
bmuller commented 8 years ago

Thanks @Reve - can you post a full proof of issue (all the code necessary to duplicate)? It could just use SQLite locally and some initial command to create a table.

Reve commented 8 years ago

I will try to see if I can isolate the issue and post a followup here.

Reve commented 8 years ago

Now it's working after a few twisted restarts. I am not sure what caused it... Thank you very much for your time.

bmuller commented 8 years ago

No worries! Let me know if you run into any other issues.

Reve commented 8 years ago

I think I finally isolated the issue. The error occurs if it's the first thing you invoke after the twisted server start. If I do a select from the DB before the save, it works. This is very strange and since it's deferred it's very hard to debug.

I did find a solution to the problem by adding the code below before the save() function is called.

validation_result = yield db_domain.isValid()
if validation_result is False:
                raise CommandParameterInvalidError(str(db_domain.errors), '50000')`

If this is the way you should validate things, I suggest that you add it to the documentation so future users will avoid this problem.

Thank you again

bmuller commented 8 years ago

Thanks for debugging @Reve - I'll take a look.

Reve commented 8 years ago

I found where the problem was. I was formating the Timestamp using the native Timestamp obtained by calling Registry.getDBAPIClass("Timestamp") and for some unknown reason it would throw a segmentation fault saying that it could not find Timestamp class/library.

Strace output:

epoll_wait(8, {{EPOLLIN, {u32=9, u64=15719776338552291337}}}, 4, 221) = 1
read(9, "xx", 8192)                     = 2
stat("/etc/localtime", {st_mode=S_IFREG|0644, st_size=2195, ...}) = 0
write(1, "2016-03-23 15:52:04+0200 [-] TWI"..., 80) = 80
stat("/etc/localtime", {st_mode=S_IFREG|0644, st_size=2195, ...}) = 0
write(3, "2016-03-23 15:52:04+0200 [-] TWI"..., 80) = 80
stat("/etc/localtime", {st_mode=S_IFREG|0644, st_size=2195, ...}) = 0
write(1, "2016-03-23 15:52:04+0200 [-] TWI"..., 65) = 65
stat("/etc/localtime", {st_mode=S_IFREG|0644, st_size=2195, ...}) = 0
write(3, "2016-03-23 15:52:04+0200 [-] TWI"..., 65) = 65
rt_sigprocmask(SIG_BLOCK, [PIPE], [], 8) = 0
sendto(14, "Q\0\0\0:BEGIN; SET TRANSACTION ISOL"..., 59, 0, NULL, 0) = 59
rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
poll([{fd=14, events=POLLIN|POLLERR}], 1, -1) = 1 ([{fd=14, revents=POLLIN}])
recvfrom(14, "C\0\0\0\nBEGIN\0C\0\0\0\10SET\0Z\0\0\0\5T", 16384, 0, NULL, NULL) = 26
rt_sigprocmask(SIG_BLOCK, [PIPE], [], 8) = 0
sendto(14, "Q\0\0\0>SELECT * FROM tags WHERE (t"..., 63, 0, NULL, 0) = 63
rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
poll([{fd=14, events=POLLIN|POLLERR}], 1, -1) = 1 ([{fd=14, revents=POLLIN}])
recvfrom(14, "T\0\0\0001\0\2id\0\0\1\26\345\0\1\0\0\0\27\0\4\377\377\377\377\0\0tag\0"..., 16384, 0, NULL, NULL) = 68
stat("/usr/lib64/python2.6/site-packages/psycopg2/Timestamp", 0x7ffc43fbd470) = -1 ENOENT (No such file or directory)
open("/usr/lib64/python2.6/site-packages/psycopg2/Timestamp.so", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/usr/lib64/python2.6/site-packages/psycopg2/Timestampmodule.so", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/usr/lib64/python2.6/site-packages/psycopg2/Timestamp.py", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/usr/lib64/python2.6/site-packages/psycopg2/Timestamp.pyc", O_RDONLY) = -1 ENOENT (No such file or directory)
stat("/etc/localtime", {st_mode=S_IFREG|0644, st_size=2195, ...}) = 0
--- SIGSEGV (Segmentation fault) @ 0 (0) ---
Process 12418 detached

I was using psycopg2 version 2.0.14. I've updated to the latest 2.6.1 and the problem disappeared. I think it should be noted somewhere which should be the minimum requirements for the dependencies.

Thank you very much once again.