Transifex Issue with authenticated proxy

[svavoi]

• GitHub URL: https://github.com/camptocamp/baselstadt_mapbs/tree/GMF2x

Running command: make -f svavoi.mk upgrade2

Result:

svavoi@bvdgdi-svdmbap3:~/mapbs$ make -f svavoi.mk upgrade2
.build/venv/bin/c2ctool upgrade  svavoi.mk --step 2
make[1]: Entering directory '/home/svavoi/mapbs'
.build/venv/bin/pot-create --config lingua-client.cfg --output mapbs/locale/mapbs-client.pot mapbs/templates/desktop.html mapbs/templates/mobile.html mapbs/templates/api/apihelp.html mapbs/templates/api/xapihelp.html mapbs/templates/match.html mapbs/templates/routing.html mapbs/templates/edit.html mapbs/templates/metadata.html mapbs/static-ngeo/js/mobile.js mapbs/static-ngeo/js/mapbsmodule.js mapbs/static-ngeo/js/desktop.js   mapbs/static-ngeo/components/contextualdata/contextualdata.html .build/config.yaml development.ini print/print-apps/mapbs/config.yaml
Traceback (most recent call last):
  File "/home/svavoi/mapbs/.build/venv/local/lib/python2.7/site-packages/c2cgeoportal/lib/lingua_extractor.py", line 126, in __call__
    exit()
  File "/home/svavoi/mapbs/.build/venv/lib/python2.7/site.py", line 403, in __call__
    raise SystemExit(code)
SystemExit: None

Traceback (most recent call last):
  File "/home/svavoi/mapbs/.build/venv/local/lib/python2.7/site-packages/c2cgeoportal/lib/lingua_extractor.py", line 126, in __call__
    exit()
  File "/home/svavoi/mapbs/.build/venv/lib/python2.7/site.py", line 403, in __call__
    raise SystemExit(code)
SystemExit: None

Error parsing ./mapbs/templates/mobile.html at line 6 column 74
Traceback (most recent call last):
  File "/home/svavoi/mapbs/.build/venv/local/lib/python2.7/site-packages/c2cgeoportal/lib/lingua_extractor.py", line 126, in __call__
    exit()
  File "/home/svavoi/mapbs/.build/venv/lib/python2.7/site.py", line 403, in __call__
    raise SystemExit(code)
SystemExit: None

Error parsing ./mapbs/templates/api/apihelp.html at line 15 column 45
Traceback (most recent call last):
  File "/home/svavoi/mapbs/.build/venv/local/lib/python2.7/site-packages/c2cgeoportal/lib/lingua_extractor.py", line 126, in __call__
    exit()
  File "/home/svavoi/mapbs/.build/venv/lib/python2.7/site.py", line 403, in __call__
    raise SystemExit(code)
SystemExit: None

Error parsing ./mapbs/templates/api/xapihelp.html at line 15 column 45
Traceback (most recent call last):
  File "/home/svavoi/mapbs/.build/venv/local/lib/python2.7/site-packages/c2cgeoportal/lib/lingua_extractor.py", line 126, in __call__
    exit()
  File "/home/svavoi/mapbs/.build/venv/lib/python2.7/site.py", line 403, in __call__
    raise SystemExit(code)
SystemExit: None

Traceback (most recent call last):
  File "/home/svavoi/mapbs/.build/venv/local/lib/python2.7/site-packages/c2cgeoportal/lib/lingua_extractor.py", line 126, in __call__
    exit()
  File "/home/svavoi/mapbs/.build/venv/lib/python2.7/site.py", line 403, in __call__
    raise SystemExit(code)
SystemExit: None

Traceback (most recent call last):
  File "/home/svavoi/mapbs/.build/venv/local/lib/python2.7/site-packages/c2cgeoportal/lib/lingua_extractor.py", line 126, in __call__
    exit()
  File "/home/svavoi/mapbs/.build/venv/lib/python2.7/site.py", line 403, in __call__
    raise SystemExit(code)
SystemExit: None

Traceback (most recent call last):
  File "/home/svavoi/mapbs/.build/venv/local/lib/python2.7/site-packages/c2cgeoportal/lib/lingua_extractor.py", line 124, in __call__
    template_adapter=MyTemplate
  File "/home/svavoi/mapbs/.build/venv/bin/bottle.py", line 3609, in template
    return TEMPLATES[tplid].render(kwargs)
  File "/home/svavoi/mapbs/.build/venv/bin/bottle.py", line 3280, in render
    return self.tpl.render(**_defaults)
  File "/home/svavoi/mapbs/.build/venv/local/lib/python2.7/site-packages/mako/template.py", line 462, in render
    return runtime._render(self, self.callable_, args, data)
  File "/home/svavoi/mapbs/.build/venv/local/lib/python2.7/site-packages/mako/runtime.py", line 838, in _render
    **_kwargs_for_callable(callable_, data))
  File "/home/svavoi/mapbs/.build/venv/local/lib/python2.7/site-packages/mako/runtime.py", line 873, in _render_context
    _exec_template(inherit, lclcontext, args=args, kwargs=kwargs)
  File "/home/svavoi/mapbs/.build/venv/local/lib/python2.7/site-packages/mako/runtime.py", line 899, in _exec_template
    callable_(context, *args, **kwargs)
  File "__mapbs_templates_metadata_html", line 24, in render_body
AttributeError: 'Undefined' object has no attribute 'bezeichnung'

Traceback (most recent call last):
  File ".build/venv/bin/pot-create", line 11, in <module>
    sys.exit(main())
  File "/home/svavoi/mapbs/.build/venv/local/lib/python2.7/site-packages/lingua/extract.py", line 330, in main
    for message in extractor(real_filename, options):
  File "/home/svavoi/mapbs/.build/venv/local/lib/python2.7/site-packages/c2cgeoportal/lib/lingua_extractor.py", line 199, in __call__
    self.env = bootstrap(filename)
  File "/home/svavoi/mapbs/.build/venv/local/lib/python2.7/site-packages/pyramid/paster.py", line 131, in bootstrap
    app = get_app(config_uri, options=options)
  File "/home/svavoi/mapbs/.build/venv/local/lib/python2.7/site-packages/pyramid/paster.py", line 31, in get_app
    global_conf=options)
  File "/home/svavoi/mapbs/.build/venv/local/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 247, in loadapp
    return loadobj(APP, uri, name=name, **kw)
  File "/home/svavoi/mapbs/.build/venv/local/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 272, in loadobj
    return context.create()
  File "/home/svavoi/mapbs/.build/venv/local/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 710, in create
    return self.object_type.invoke(self)
  File "/home/svavoi/mapbs/.build/venv/local/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 203, in invoke
    app = context.app_context.create()
  File "/home/svavoi/mapbs/.build/venv/local/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 710, in create
    return self.object_type.invoke(self)
  File "/home/svavoi/mapbs/.build/venv/local/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 146, in invoke
    return fix_call(context.object, context.global_conf, **context.local_conf)
  File "/home/svavoi/mapbs/.build/venv/local/lib/python2.7/site-packages/paste/deploy/util.py", line 55, in fix_call
    val = callable(*args, **kw)
  File "/home/svavoi/mapbs/mapbs/__init__.py", line 66, in main
    cube_engine = create_engine(settings.get('cube_connection').get('url'))
AttributeError: 'NoneType' object has no attribute 'get'
CONST_Makefile:701: recipe for target 'mapbs/locale/mapbs-client.pot' failed
make[1]: *** [mapbs/locale/mapbs-client.pot] Error 1
make[1]: Leaving directory '/home/svavoi/mapbs'
Traceback (most recent call last):
  File ".build/venv/bin/c2ctool", line 11, in <module>
    sys.exit(main())
  File "/home/svavoi/mapbs/.build/venv/local/lib/python2.7/site-packages/c2cgeoportal/scripts/c2ctool.py", line 98, in main
    c2ctool.upgrade()
  File "/home/svavoi/mapbs/.build/venv/local/lib/python2.7/site-packages/c2cgeoportal/scripts/c2ctool.py", line 218, in upgrade
    self.step2()
  File "/home/svavoi/mapbs/.build/venv/local/lib/python2.7/site-packages/c2cgeoportal/scripts/c2ctool.py", line 383, in step2
    self.step3()
  File "/home/svavoi/mapbs/.build/venv/local/lib/python2.7/site-packages/c2cgeoportal/scripts/c2ctool.py", line 428, in step3
    self.step4()
  File "/home/svavoi/mapbs/.build/venv/local/lib/python2.7/site-packages/c2cgeoportal/scripts/c2ctool.py", line 442, in step4
    check_call(["make", "-f", self.options.file, "build"])
  File "/usr/lib/python2.7/subprocess.py", line 540, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['make', '-f', 'svavoi.mk', 'build']' returned non-zero exit status 2
CONST_Makefile:1248: recipe for target 'upgrade2' failed
make: *** [upgrade2] Error 1

My assumption is that the DB connections I defined in the __init__.py file are not valid anymore. Do I need to define them differently for GMF 2.x? https://github.com/camptocamp/baselstadt_mapbs/blob/GMF2x/mapbs/__init__.py#L42-L49

[svavoi]

How they are defined at the moment:

def main(global_config, **settings):
    """ This function returns a Pyramid WSGI application.
    """
    settings.update(yaml.load(file(settings.get("app.cfg"))))
    config = Configurator(root_factory=Root, settings=settings,
            locale_negotiator=locale_negotiator,
            authentication_policy=create_authentication(settings))

    # create a connection to the cube database
    cube_engine = create_engine(settings.get('cube_connection').get('url'))
    sqlahelper.add_engine(cube_engine, 'cube')
    # create a connection to the digi database
    digi_engine = create_engine(settings.get('digi_connection').get('url'))
    sqlahelper.add_engine(digi_engine, 'digi')
    # create a connection to the pmt database
    pmt_engine = create_engine(settings.get('pmt_connection').get('url'))
    sqlahelper.add_engine(pmt_engine, 'pmt')

[ger-benjamin]

I thinks that I can't see the vars_secure.yaml file but can you confirm that cube_connection is defined ?

Also, I'm not really sur if this line is right: settings.update(yaml.load(file(settings.get("app.cfg"))))

To test, you can add some print(<value>) betweens the lines of the __init__.py file to see in the error log file if you get what you want. That can help to precise where is the issue. (Like

• print(settings.get('cube_connection')
• or print(file(settings.get("app.cfg"))))

Can you verify / test and give me more informations ?

[svavoi]

The cube_connetion is defined in the vars_mapbs.yaml file

The vars_secure.yaml looks like this:

extends: vars_mapbs.yaml

vars:
    # database password
    dbpassword: password

    # cube db password
    cube_dbpassword: password

...

I tried to add the print statements to the __init__.py file but I don't know where I can see them in the logs. In the apache logs ( /var/www/vhosts/mapbs/logs) are no entries because I can not build the application yet. Can you tell me where to look?

[ger-benjamin]

Hello, oh yeah sorry, you can see your "print(...)" directly in what is displayed during the run of the command (not in the log file). (Obviously, it need to passes in the "main" method of the __init__.py).

That can be hard to see where is the the printed result. You can print some characters to easy find your result So, doing something like that:

in the __init__.py file's main method, add somewhere after the settings.update(yaml.load(file(settings.get("app.cfg")))) line, add:

print('!!!!!!!!!!!')
print(settings.get('cube_connection'))
print(file(settings.get("app.cfg")))
print('!!!!!!!!!!!')

I just hope that it passes in this main method each time you relaunch the "upgrade2" command. Else, you can try a build (or clean then build)

[svavoi]

Oh yes thanks, now I was able to find it. This is what I get:

!!!!!!!!!!!
None
<open file '/home/svavoi/mapbs/.build/config.yaml', mode 'r' at 0x7fe8a609e8a0>
!!!!!!!!!!!

In the /home/svavoi/mapbs/.build/config.yaml file I can find the cube_connection:

cube_connection: {url: 'postgresql://...'}

[ger-benjamin]

Ok, I think that your cube_connection in in the vars object. So, if your vars file (and .build/config.yaml file) look like this:

...
vars:
  ...
  cube_connection:
    url: abcde
  ...

You can access to the url by writing:

cube_connection = create_engine(settings.get(`vars`).get('cube_connection').get('url'))

Or:

vars_params = settings.get(`vars`)
cube_connection = create_engine(vars_params.get('cube_connection').get('url'))

[svavoi]

Thanks! I can get now the cube_connection. But I still have this error and some new ones:

Traceback (most recent call last):
  File "/home/svavoi/mapbs/.build/venv/local/lib/python2.7/site-packages/c2cgeoportal/lib/lingua_extractor.py", line 124, in __call__
    template_adapter=MyTemplate
  File "/home/svavoi/mapbs/.build/venv/bin/bottle.py", line 3609, in template
    return TEMPLATES[tplid].render(kwargs)
  File "/home/svavoi/mapbs/.build/venv/bin/bottle.py", line 3280, in render
    return self.tpl.render(**_defaults)
  File "/home/svavoi/mapbs/.build/venv/local/lib/python2.7/site-packages/mako/template.py", line 462, in render
    return runtime._render(self, self.callable_, args, data)
  File "/home/svavoi/mapbs/.build/venv/local/lib/python2.7/site-packages/mako/runtime.py", line 838, in _render
    **_kwargs_for_callable(callable_, data))
  File "/home/svavoi/mapbs/.build/venv/local/lib/python2.7/site-packages/mako/runtime.py", line 873, in _render_context
    _exec_template(inherit, lclcontext, args=args, kwargs=kwargs)
  File "/home/svavoi/mapbs/.build/venv/local/lib/python2.7/site-packages/mako/runtime.py", line 899, in _exec_template
    callable_(context, *args, **kwargs)
  File "__mapbs_templates_metadata_html", line 24, in render_body
AttributeError: 'Undefined' object has no attribute 'bezeichnung'
ERROR: The database is probably not up to date (should be ignored when happen during the upgrade)
(psycopg2.ProgrammingError) relation "main.layer_wms" does not exist
LINE 2: ...in.layer ON main.treeitem.id = main.layer.id JOIN main.layer...
                                                             ^
 [SQL: 'SELECT main.treeitem.type AS main_treeitem_type, main.layer_wms.id AS main_layer_wms_id, main.layer.id AS main_layer_id, main.treeitem.id AS main_treeitem_id, main.treeitem.name AS main_treeitem_name, main.treeitem.metadata_url AS main_treeitem_metadata_url, main.treeitem.description AS main_treeitem_description, main.layer.public AS main_layer_public, main.layer.geo_table AS main_layer_geo_table, main.layer.exclude_properties AS main_layer_exclude_properties, main.layer_wms.ogc_server_id AS main_layer_wms_ogc_server_id, main.layer_wms.layer AS main_layer_wms_layer, main.layer_wms.style AS main_layer_wms_style, main.layer_wms.time_mode AS main_layer_wms_time_mode, main.layer_wms.time_widget AS main_layer_wms_time_widget \nFROM main.treeitem JOIN main.layer ON main.treeitem.id = main.layer.id JOIN main.layer_wms ON main.layer.id = main.layer_wms.id']

I defined a model using this connection in the models.py:

cube_engine = sqlahelper.get_engine('cube')
CubeSession = orm.scoped_session(orm.sessionmaker(extension=[sqlahelper._zte]))
CubeSession.configure(bind=cube_engine)
CubeBase = declarative_base()

class Metadata_Datensatz_MapBS(CubeBase):
    __tablename__ = "view_metadaten_mapbs"
    __table_args__ = {"schema": "ds"}
    mapbs_id = Column(Integer, primary_key=True)
    bezeichnung = Column(Text)
    datensatz_bez = Column(Text)
...

and using it in the metadata.py here https://github.com/camptocamp/baselstadt_mapbs/blob/GMF2x/mapbs/views/metadata.py. Is there something I should change in the definition?

[ger-benjamin]

The error is relation "main.layer_wms" does not exist.

At the end of the upgrade1, you should have seen the message that ask you to apply the manual migration steps based on what is in the CONST_CHANGELOG.txt. To you do this ? Do you have run the point 12 :

https://github.com/camptocamp/baselstadt_mapbs/blob/GMF2x/CONST_CHANGELOG.txt#L195-L199

The command should, amongst others, create and migrate your data in the table main.wms_layers. (From 1.6.x to 2.x, there is several major changes in the database in order to manage the layers.)

[svavoi]

No indeed I didn't run the pre upgrade yet because I was not sure if I can do it without running point 6 of version 2.0.0 before: https://github.com/camptocamp/baselstadt_mapbs/blob/GMF2x/CONST_CHANGELOG.txt#L268 Do I need to run this script as well?

Point 12 is now done. My next error is the following:

svavoi@bvdgdi-svdmbap3:~/mapbs$ make -f svavoi.mk upgrade2
.build/venv/bin/c2ctool upgrade  svavoi.mk --step 2
make[1]: Entering directory '/home/svavoi/mapbs'
mkdir -p .build/locale/de/LC_MESSAGES/
.build/venv/bin/tx pull -l de --force
MaxRetryError: HTTPSConnectionPool(host='www.transifex.com', port=443): Max retries exceeded with url: /api/2/project/ngeo/resource/ngeo-2_1/stats/ (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x7effa4520450>: Failed to establish a new connection: [Errno 101] Network is unreachable',))
CONST_Makefile:751: recipe for target '.build/locale/de/LC_MESSAGES/gmf.po' failed
make[1]: *** [.build/locale/de/LC_MESSAGES/gmf.po] Error 1
make[1]: Leaving directory '/home/svavoi/mapbs'
Traceback (most recent call last):
  File ".build/venv/bin/c2ctool", line 11, in <module>
    sys.exit(main())
  File "/home/svavoi/mapbs/.build/venv/local/lib/python2.7/site-packages/c2cgeoportal/scripts/c2ctool.py", line 98, in main
    c2ctool.upgrade()
  File "/home/svavoi/mapbs/.build/venv/local/lib/python2.7/site-packages/c2cgeoportal/scripts/c2ctool.py", line 218, in upgrade
    self.step2()
  File "/home/svavoi/mapbs/.build/venv/local/lib/python2.7/site-packages/c2cgeoportal/scripts/c2ctool.py", line 383, in step2
    self.step3()
  File "/home/svavoi/mapbs/.build/venv/local/lib/python2.7/site-packages/c2cgeoportal/scripts/c2ctool.py", line 428, in step3
    self.step4()
  File "/home/svavoi/mapbs/.build/venv/local/lib/python2.7/site-packages/c2cgeoportal/scripts/c2ctool.py", line 442, in step4
    check_call(["make", "-f", self.options.file, "build"])
  File "/usr/lib/python2.7/subprocess.py", line 540, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['make', '-f', 'svavoi.mk', 'build']' returned non-zero exit status 2
CONST_Makefile:1248: recipe for target 'upgrade2' failed
make: *** [upgrade2] Error 1

[ger-benjamin]

Yes, if you target the 2.1.0, you must apply all the points in the changelog directly

Failed to establish a new connection: [Errno 101] Network is unreachable

Can you try again ? We use www.transifex.com (not a c2c service) to manage our c2cgeoportal translation. It seems to be unreachable. Perhaps the problem was in the transifex side... Or do you limit connections on externals servers ?

[svavoi]

I tried again but it did not work. I guess the problem is that our http_proxy doesn't allow the connection. Is there a way to configure a proxy that should be used to establish this connection?

[ger-benjamin]

Sorry, I really don't know how to do that on your servers. Can you ask one of your sysadmin ? (I don't think that we manage your servers, is that right ?)

[svavoi]

That's right you do not manage our servers. I already asked them, but they say that the proxy is set but not taken into account in this step. The proxy to use is configured here: https://github.com/camptocamp/baselstadt_mapbs/blob/GMF2x/apache/application.wsgi.mako#L13 How can I make sure that it is used?

[ger-benjamin]

What do you get if you launch the command curl https://www.transifex.com:443 ? (you should get an html page)

[svavoi]

The command works. I get a html page.

svavoi@bvdgdi-svdmbap3:~/mapbs$ curl https://www.transifex.com:443
<!doctype html>
<html class="site no-js" lang="en">

<head>
  <!--[if lt IE 9]>
    <script src="//cdnjs.cloudflare.com/ajax/libs/html5shiv/3.7.2/html5shiv.min.js"></script>
  <![endif]-->
...
...

[ger-benjamin]

I'm out of idea. That's really strange. The code seems right and you can curl it, so it shouldn't be any problem. And I'm not here tomorrow. Perhaps @sbrunner will have more informations about that but he is already in Holidays.

[svavoi]

I tried again, this time I get a different error:

.build/venv/bin/c2ctool upgrade  svavoi.mk --step 2
make[1]: Entering directory '/home/svavoi/mapbs'
.build/venv/bin/pot-create --config lingua-client.cfg --output mapbs/locale/mapbs-client.pot mapbs/templates/desktop.html mapbs/templates/mobile.html mapbs/templates/api/apihelp.                                                           html mapbs/templates/api/xapihelp.html mapbs/templates/match.html mapbs/templates/routing.html mapbs/templates/edit.html mapbs/templates/metadata.html mapbs/static-ngeo/js/mobile                                                           .js mapbs/static-ngeo/js/mapbsmodule.js mapbs/static-ngeo/js/desktop.js   mapbs/static-ngeo/components/contextualdata/contextualdata.html .build/config.yaml development.ini print                                                           /print-apps/mapbs/config.yaml
Traceback (most recent call last):
  File "/home/svavoi/mapbs/.build/venv/local/lib/python2.7/site-packages/c2cgeoportal/lib/lingua_extractor.py", line 126, in __call__
    exit()
  File "/home/svavoi/mapbs/.build/venv/lib/python2.7/site.py", line 403, in __call__
    raise SystemExit(code)
SystemExit: None

Traceback (most recent call last):
  File "/home/svavoi/mapbs/.build/venv/local/lib/python2.7/site-packages/c2cgeoportal/lib/lingua_extractor.py", line 126, in __call__
    exit()
  File "/home/svavoi/mapbs/.build/venv/lib/python2.7/site.py", line 403, in __call__
    raise SystemExit(code)
SystemExit: None

Error parsing ./mapbs/templates/mobile.html at line 6 column 74
Traceback (most recent call last):
  File "/home/svavoi/mapbs/.build/venv/local/lib/python2.7/site-packages/c2cgeoportal/lib/lingua_extractor.py", line 126, in __call__
    exit()
  File "/home/svavoi/mapbs/.build/venv/lib/python2.7/site.py", line 403, in __call__
    raise SystemExit(code)
SystemExit: None

Error parsing ./mapbs/templates/api/apihelp.html at line 15 column 45
Traceback (most recent call last):
  File "/home/svavoi/mapbs/.build/venv/local/lib/python2.7/site-packages/c2cgeoportal/lib/lingua_extractor.py", line 126, in __call__
    exit()
  File "/home/svavoi/mapbs/.build/venv/lib/python2.7/site.py", line 403, in __call__
    raise SystemExit(code)
SystemExit: None

Error parsing ./mapbs/templates/api/xapihelp.html at line 15 column 45
Traceback (most recent call last):
  File "/home/svavoi/mapbs/.build/venv/local/lib/python2.7/site-packages/c2cgeoportal/lib/lingua_extractor.py", line 126, in __call__
    exit()
  File "/home/svavoi/mapbs/.build/venv/lib/python2.7/site.py", line 403, in __call__
    raise SystemExit(code)
SystemExit: None

Traceback (most recent call last):
  File "/home/svavoi/mapbs/.build/venv/local/lib/python2.7/site-packages/c2cgeoportal/lib/lingua_extractor.py", line 126, in __call__
    exit()
  File "/home/svavoi/mapbs/.build/venv/lib/python2.7/site.py", line 403, in __call__
    raise SystemExit(code)
SystemExit: None

Traceback (most recent call last):
  File "/home/svavoi/mapbs/.build/venv/local/lib/python2.7/site-packages/c2cgeoportal/lib/lingua_extractor.py", line 126, in __call__
    exit()
  File "/home/svavoi/mapbs/.build/venv/lib/python2.7/site.py", line 403, in __call__
    raise SystemExit(code)
SystemExit: None

Traceback (most recent call last):
  File "/home/svavoi/mapbs/.build/venv/local/lib/python2.7/site-packages/c2cgeoportal/lib/lingua_extractor.py", line 124, in __call__
    template_adapter=MyTemplate
  File "/home/svavoi/mapbs/.build/venv/bin/bottle.py", line 3609, in template
    return TEMPLATES[tplid].render(kwargs)
  File "/home/svavoi/mapbs/.build/venv/bin/bottle.py", line 3280, in render
    return self.tpl.render(**_defaults)
  File "/home/svavoi/mapbs/.build/venv/local/lib/python2.7/site-packages/mako/template.py", line 462, in render
    return runtime._render(self, self.callable_, args, data)
  File "/home/svavoi/mapbs/.build/venv/local/lib/python2.7/site-packages/mako/runtime.py", line 838, in _render
    **_kwargs_for_callable(callable_, data))
  File "/home/svavoi/mapbs/.build/venv/local/lib/python2.7/site-packages/mako/runtime.py", line 873, in _render_context
    _exec_template(inherit, lclcontext, args=args, kwargs=kwargs)
  File "/home/svavoi/mapbs/.build/venv/local/lib/python2.7/site-packages/mako/runtime.py", line 899, in _exec_template
    callable_(context, *args, **kwargs)
  File "__mapbs_templates_metadata_html", line 24, in render_body
AttributeError: 'Undefined' object has no attribute 'bezeichnung'

Get DescribeFeatureType for url: http://localhost/svavoi/mapserv?VERSION=1.1.0&REQUEST=DescribeFeatureType&SERVICE=WFS
DescribeFeatureType from URL http://localhost/svavoi/mapserv?VERSION=1.1.0&REQUEST=DescribeFeatureType&SERVICE=WFS return the error: 404 Not Found
No such table 'rstg_baumkataster.baumgeo' for layer 'BA_Baumkataster_baum'.
CONST_Makefile:701: recipe for target 'mapbs/locale/mapbs-client.pot' failed
make[1]: *** [mapbs/locale/mapbs-client.pot] Error 1
make[1]: Leaving directory '/home/svavoi/mapbs'
Traceback (most recent call last):
  File ".build/venv/bin/c2ctool", line 11, in <module>
    sys.exit(main())
  File "/home/svavoi/mapbs/.build/venv/local/lib/python2.7/site-packages/c2cgeoportal/scripts/c2ctool.py", line 98, in main
    c2ctool.upgrade()
  File "/home/svavoi/mapbs/.build/venv/local/lib/python2.7/site-packages/c2cgeoportal/scripts/c2ctool.py", line 218, in upgrade
    self.step2()
  File "/home/svavoi/mapbs/.build/venv/local/lib/python2.7/site-packages/c2cgeoportal/scripts/c2ctool.py", line 383, in step2
    self.step3()
  File "/home/svavoi/mapbs/.build/venv/local/lib/python2.7/site-packages/c2cgeoportal/scripts/c2ctool.py", line 428, in step3
    self.step4()
  File "/home/svavoi/mapbs/.build/venv/local/lib/python2.7/site-packages/c2cgeoportal/scripts/c2ctool.py", line 442, in step4
    check_call(["make", "-f", self.options.file, "build"])
  File "/usr/lib/python2.7/subprocess.py", line 540, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['make', '-f', 'svavoi.mk', 'build']' returned non-zero exit status 2
CONST_Makefile:1248: recipe for target 'upgrade2' failed
make: *** [upgrade2] Error 1

I don't understand why this table can not be found:

No such table 'rstg_baumkataster.baumgeo' for layer 'BA_Baumkataster_baum'.

The table is there and the db connection seems right to me...

[ger-benjamin]

Can you verify if the db CONNECTION settings are set correctly for the layer BA_Baumkataster_baum ? (in your .map, and not in the ?) Also, in the database (the one specified in the CONNECTION parameter), can you try to do a SELECT * FROM rstg_baumkataster.baumgeo;

You can also try to open the link: http://<your_dns>/svavoi/mapserv_proxy?VERSION=1.1.0&REQUEST=DescribeFeatureType&SERVICE=WFS in your browser. You should got the same error but perhaps you will get somethings more in the log (in the apache log this time)

[svavoi]

I checked the db CONNECTION setting again and it is right. In the database I have the permission to select the data in this table.

But I can't open this link http://<your_dns>/svavoi/mapserv_proxy?VERSION=1.1.0&REQUEST=DescribeFeatureType&SERVICE=WFS or this http://<your_dns>/svavoi/wsgi/mapserv_proxy?VERSION=1.1.0&REQUEST=DescribeFeatureType&SERVICE=WFS I always get a 404 Error.

Also if I do a wget http://localhost/svavoi/wsgi/mapserv_proxy?VERSION=1.1.0&REQUEST=DescribeFeatureType&SERVICE=WFS

I get the following error:

svavoi@bvdgdi-svdmbap3:~/mapbs$ wget http://localhost/svavoi/wsgi/mapserv_proxy?VERSION=1.1.0&REQUEST=DescribeFeatureType&SERVICE=WFS
[1] 1337
[2] 1338
svavoi@bvdgdi-svdmbap3:~/mapbs$ --2017-01-05 15:16:27--  http://localhost/svavoi/wsgi/mapserv_proxy?VERSION=1.1.0
Resolving localhost (localhost)... ::1, 127.0.0.1
Connecting to localhost (localhost)|::1|:80... connected.
HTTP request sent, awaiting response... 404 Not Found
2017-01-05 15:16:27 ERROR 404: Not Found.

[ger-benjamin]

And you don't have mad a make <...> clean ? Right ?

[svavoi]

I did run a make <...> clean and rerun a make <...> build but it didn't help. Still the same error.

[ger-benjamin]

Oh I've don't seen that we also have a 404 error in your previous error (not that with you test, the localhost can't work, you must use the external host. And perhaps in https ?) Anyway, that just to collect some information but that's not the real issue.

Do you give all the right to www-data for the schema rstg_baumkataster and table baumgeo ? Otherwise you can comments this layer for now, just to see if any other layers have this error ? (I'm in search of more clues for this issue...)

[svavoi]

The www-data user has USAGE right on the schema and SELECT, INSERT, UPDATE, DELETE rights on the table baumgeo.

What I tried:

• in mapserver/c2cgeoportal.map.mako I comment the INCLUDE statement#INCLUDE "include/Fachdaten/BA_Baumkataster_digi.map"
• comment out the layer in mapserver/include/Fachdaten/BA_Baumkataster_digi.map.mako
• run amake <...> clean and make <...> build

But nothing helped. The error stays the same.

[sbrunner]

The error don't come from the mapfile but from the geotable in the database ...

[svavoi]

I had a look in the database and realized that in the main.layer table all the entries are twice in the table.

select count(*) from main.layer; => 2312
select count(*) from main.layerv1; => 1156

select a.id, b.id as layerv1_id, a.geo_table, a.exclude_properties, b.layer as layerv1, b.layer_type, c.layer as layer_wms
from main.layer a
left join main.layerv1 b on a.id = b.id
left join main.layer_wms c on a.id = c.id
where a.geo_table like 'rstg_baumkataster.baumgeo';

Is this normal after the pre upgrade of the database?

This geo_table rstg_baumkataster.baumgeo is not located in the same database as I run the upgrade on. Is this a problem?

[sbrunner]

Is this normal after the pre upgrade of the database?

If you run the themev1tov2 script, yes

This geo_table rstg_baumkataster.baumgeo is not located in the same database as I run the upgrade on. Is this a problem?

Effectively it can't work ...

[svavoi]

Hmmm, so what do you suggest?

[eleu]

With the version 2, we now get the strings to translate (i18n) in the edit form from the data table of the edit layer. Therefore, the data table has to be in the same data base.

Your issue will probably be solved if you could put your data table of the edit layer into your current database, with just its structure and without data.

[svavoi]

I put the structure of all the external tables in the main database and this helped. But now I get again the transifex problem:

 upgrade  svavoi.mk --step 2
mkdir -p .build/locale/de/LC_MESSAGES/
.build/venv/bin/tx pull -l de --force
MaxRetryError: HTTPSConnectionPool(host='www.transifex.com', port=443): Max retries exceeded with url: /api/2/project/ngeo/resource/ngeo-2_1/stats/ (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x7f6148565490>: Failed to establish a new connection: [Errno 101] Network is unreachable',))
CONST_Makefile:751: recipe for target '.build/locale/de/LC_MESSAGES/gmf.po' failed
make: *** [.build/locale/de/LC_MESSAGES/gmf.po] Error 1
Traceback (most recent call last):
  File ".build/venv/bin/c2ctool", line 11, in <module>
    sys.exit(main())
  File "/home/svavoi/mapbs/.build/venv/local/lib/python2.7/site-packages/c2cgeoportal/scripts/c2ctool.py", line 98, in main
    c2ctool.upgrade()
  File "/home/svavoi/mapbs/.build/venv/local/lib/python2.7/site-packages/c2cgeoportal/scripts/c2ctool.py", line 218, in upgrade
    self.step2()
  File "/home/svavoi/mapbs/.build/venv/local/lib/python2.7/site-packages/c2cgeoportal/scripts/c2ctool.py", line 383, in step2
    self.step3()
  File "/home/svavoi/mapbs/.build/venv/local/lib/python2.7/site-packages/c2cgeoportal/scripts/c2ctool.py", line 428, in step3
    self.step4()
  File "/home/svavoi/mapbs/.build/venv/local/lib/python2.7/site-packages/c2cgeoportal/scripts/c2ctool.py", line 442, in step4
    check_call(["make", "-f", self.options.file, "build"])
  File "/usr/lib/python2.7/subprocess.py", line 540, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['make', '-f', 'svavoi.mk', 'build']' returned non-zero exit status 2

curl https://www.transifex.com:443 and curl http://www.transifex.com:443 works fine, but not curl www.transifex.com:443

Can I change somewhere how the connection will be done? It should use already https://, or not? https://github.com/camptocamp/baselstadt_mapbs/blob/GMF2x/CONST_Makefile#L248

[sbrunner]

You can try to set an other value...

[svavoi]

Where can I do this? The error message doesn't change when I change the value here: https://github.com/camptocamp/baselstadt_mapbs/blob/GMF2x/CONST_Makefile#L248 No "https://" is added...

MaxRetryError: HTTPSConnectionPool(host='www.transifex.com', port=443)

[sbrunner]

I just see that this part of code change, than can you test it:

.build/venv/bin/pip install transifex-client==0.12.3
make -f svavoi.mk .build/locale/de/LC_MESSAGES/gmf.po

Thanks :-)

[svavoi]

Installation was successfull but I get following error when I run the make:

 make -f svavoi.mk .build/locale/de/LC_MESSAGES/gmf.po
mkdir -p .build/locale/de/LC_MESSAGES/
.build/venv/bin/tx pull -l de --force
MaxRetryError: HTTPSConnectionPool(host='www.transifex.com', port=443): Max retries exceeded with url: /api/2/project/ngeo/resource/ngeo-2_1/stats/ (Caused by ProxyError('Cannot connect to proxy.', error('Tunnel connection failed: 407 Proxy Authentication Required',)))
CONST_Makefile:751: recipe for target '.build/locale/de/LC_MESSAGES/gmf.po' failed
make: *** [.build/locale/de/LC_MESSAGES/gmf.po] Error 1

[sbrunner]

How is your authenticated proxy is configured? Is your username and password configured in the proxy url?

[svavoi]

I did export my proxy settings like this:

export http_proxy=http://<user>:<passwort>@pvproi45.bs.ch:3128 
export https_proxy=https://<user>:<passwort>@pvproi45.bs.ch:3128 

and it is set here as well: https://github.com/camptocamp/baselstadt_mapbs/blob/GMF2x/apache/application.wsgi.mako#L13

Is there anywhere else I should put it?

[sbrunner]

Can you test my changes by doing:

.build/venv/bin/pip install git+https://github.com/sbrunner/transifex-client.git@authenticated-proxy
make -f svavoi.mk .build/locale/de/LC_MESSAGES/gmf.po

Thanks in advance :-)

[svavoi]

This is the result:

svavoi@bvdgdi-svdmbap3:~/mapbs$ .build/venv/bin/pip install git+https://github.com/sbrunner/transifex-client.git@authenticated-proxy
Collecting git+https://github.com/sbrunner/transifex-client.git@authenticated-proxy
  Cloning https://github.com/sbrunner/transifex-client.git (to authenticated-proxy) to /tmp/pip-HNI5UO-build
  Requirement already satisfied (use --upgrade to upgrade): transifex-client==0.12.3 from git+https://github.com/sbrunner/transifex-client.git@authenticated-proxy in ./.build/venv/lib/python2.7/site-packages
Requirement already satisfied: urllib3 in ./.build/venv/lib/python2.7/site-packages (from transifex-client==0.12.3)
Requirement already satisfied: six in ./.build/venv/lib/python2.7/site-packages (from transifex-client==0.12.3)
svavoi@bvdgdi-svdmbap3:~/mapbs$ make -f svavoi.mk .build/locale/de/LC_MESSAGES/gmf.po
mkdir -p .build/locale/de/LC_MESSAGES/
.build/venv/bin/tx pull -l de --force
MaxRetryError: HTTPSConnectionPool(host='www.transifex.com', port=443): Max retries exceeded with url: /api/2/project/ngeo/resource/ngeo-2_1/stats/ (Caused by ProxyError('Cannot connect to proxy.', error('Tunnel connection failed: 407 Proxy Authentication Required',)))
CONST_Makefile:751: recipe for target '.build/locale/de/LC_MESSAGES/gmf.po' failed
make: *** [.build/locale/de/LC_MESSAGES/gmf.po] Error 1

[sbrunner]

I suspect that the pip install don't do anything than tray again with:

.build/venv/bin/pip uninstall transifex-client
.build/venv/bin/pip install git+https://github.com/sbrunner/transifex-client.git@authenticated-proxy
make -f svavoi.mk .build/locale/de/LC_MESSAGES/gmf.po

Thanks in advance :-)

[svavoi]

still doesn't work...

svavoi@bvdgdi-svdmbap3:~/mapbs$ .build/venv/bin/pip uninstall transifex-client
Uninstalling transifex-client-0.12.3:
  /home/svavoi/mapbs/.build/venv/bin/tx
  /home/svavoi/mapbs/.build/venv/lib/python2.7/site-packages/transifex_client-0.12.3-py2.7.egg-info
  /home/svavoi/mapbs/.build/venv/lib/python2.7/site-packages/txclib/__init__.py
  /home/svavoi/mapbs/.build/venv/lib/python2.7/site-packages/txclib/__init__.pyc
  /home/svavoi/mapbs/.build/venv/lib/python2.7/site-packages/txclib/__main__.py
  /home/svavoi/mapbs/.build/venv/lib/python2.7/site-packages/txclib/__main__.pyc
  /home/svavoi/mapbs/.build/venv/lib/python2.7/site-packages/txclib/cacert.pem
  /home/svavoi/mapbs/.build/venv/lib/python2.7/site-packages/txclib/cmdline.py
  /home/svavoi/mapbs/.build/venv/lib/python2.7/site-packages/txclib/cmdline.pyc
  /home/svavoi/mapbs/.build/venv/lib/python2.7/site-packages/txclib/commands.py
  /home/svavoi/mapbs/.build/venv/lib/python2.7/site-packages/txclib/commands.pyc
  /home/svavoi/mapbs/.build/venv/lib/python2.7/site-packages/txclib/config.py
  /home/svavoi/mapbs/.build/venv/lib/python2.7/site-packages/txclib/config.pyc
  /home/svavoi/mapbs/.build/venv/lib/python2.7/site-packages/txclib/exceptions.py
  /home/svavoi/mapbs/.build/venv/lib/python2.7/site-packages/txclib/exceptions.pyc
  /home/svavoi/mapbs/.build/venv/lib/python2.7/site-packages/txclib/log.py
  /home/svavoi/mapbs/.build/venv/lib/python2.7/site-packages/txclib/log.pyc
  /home/svavoi/mapbs/.build/venv/lib/python2.7/site-packages/txclib/parsers.py
  /home/svavoi/mapbs/.build/venv/lib/python2.7/site-packages/txclib/parsers.pyc
  /home/svavoi/mapbs/.build/venv/lib/python2.7/site-packages/txclib/paths.py
  /home/svavoi/mapbs/.build/venv/lib/python2.7/site-packages/txclib/paths.pyc
  /home/svavoi/mapbs/.build/venv/lib/python2.7/site-packages/txclib/processors.py
  /home/svavoi/mapbs/.build/venv/lib/python2.7/site-packages/txclib/processors.pyc
  /home/svavoi/mapbs/.build/venv/lib/python2.7/site-packages/txclib/project.py
  /home/svavoi/mapbs/.build/venv/lib/python2.7/site-packages/txclib/project.pyc
  /home/svavoi/mapbs/.build/venv/lib/python2.7/site-packages/txclib/urls.py
  /home/svavoi/mapbs/.build/venv/lib/python2.7/site-packages/txclib/urls.pyc
  /home/svavoi/mapbs/.build/venv/lib/python2.7/site-packages/txclib/utils.py
  /home/svavoi/mapbs/.build/venv/lib/python2.7/site-packages/txclib/utils.pyc
  /home/svavoi/mapbs/.build/venv/lib/python2.7/site-packages/txclib/web.py
  /home/svavoi/mapbs/.build/venv/lib/python2.7/site-packages/txclib/web.pyc
Proceed (y/n)? y
  Successfully uninstalled transifex-client-0.12.3
svavoi@bvdgdi-svdmbap3:~/mapbs$ .build/venv/bin/pip install git+https://github.com/sbrunner/transifex-client.git@authenticated-proxy
Collecting git+https://github.com/sbrunner/transifex-client.git@authenticated-proxy
  Cloning https://github.com/sbrunner/transifex-client.git (to authenticated-proxy) to /tmp/pip-PV7feC-build
Requirement already satisfied: urllib3 in ./.build/venv/lib/python2.7/site-packages (from transifex-client==0.12.3)
Requirement already satisfied: six in ./.build/venv/lib/python2.7/site-packages (from transifex-client==0.12.3)
Installing collected packages: transifex-client
  Running setup.py install for transifex-client ... done
Successfully installed transifex-client-0.12.3
svavoi@bvdgdi-svdmbap3:~/mapbs$ make -f svavoi.mk .build/locale/de/LC_MESSAGES/gmf.po
mkdir -p .build/locale/de/LC_MESSAGES/
.build/venv/bin/tx pull -l de --force
MaxRetryError: HTTPSConnectionPool(host='www.transifex.com', port=443): Max retries exceeded with url: /api/2/project/ngeo/resource/ngeo-2_1/stats/ (Caused by ProxyError('Cannot connect to proxy.', error('Tunnel connection failed: 407 Proxy Authentication Required',)))
CONST_Makefile:751: recipe for target '.build/locale/de/LC_MESSAGES/gmf.po' failed
make: *** [.build/locale/de/LC_MESSAGES/gmf.po] Error 1

[ebelo]

@svavoi the authenticated proxy needs basic auth with username/password, right?

Currently there is this limitation in the tx client: https://docs.transifex.com/client/introduction#help (no auth). I'm going to open an issue on their github, but would like to be sure on the auth method

https://github.com/transifex/transifex-client/issues/166

[svavoi]

Yes, that's right. We need a basic auth with username/password like: export http_proxy=http://<user>:<passwort>@<proxy>:<port>

Is there a way to continue with the upgrade without connecting to transifex right now?

[sbrunner]

Yes you can download the po files from here: https://www.transifex.com/camptocamp/ngeo/language/de/ (ngeo-2_1 and gmf-2_1)

[svavoi]

I don't seem to have access to this folder. Can you provide a more detailed instruction please?

[sbrunner]

• Open https://www.transifex.com/signin/, Login with whatever you want
• Open https://www.transifex.com/camptocamp/ngeo/language/de/
• Click on ngeo-2_1
• Click on Download the file to use it (for me it's in French :-) )
• Do the scale for gmf-2_1

[sbrunner]

Transifex requires that your logged in to download the file but you don't have to have some special rights :-)

[ger-benjamin]

I think that the transifex doesn't use the work of SBrunner. Can you try with: export https_proxy=https://<user>:<passwort>@<proxy>:<port> (with a "s" at both "http" words (key and url))

If you have done a make -f <mk file> clean, do again these steps after the first build attempt:

make -f <mk file> build
.build/venv/bin/pip uninstall transifex-client
.build/venv/bin/pip install git+https://github.com/sbrunner/transifex-client.git@authenticated-proxy
make -f svavoi.mk .build/locale/de/LC_MESSAGES/gmf.po

Update: Oh, sorry, I've seen that you already export these values (There are always existing, right ?). So, that should works... forgot this comment

[ger-benjamin]

Update: I've found a little issue and sbrunner has updated its code. Can you try again ? You will must reinstall the transifex-client to download the change. (

• Verify that you export the https_proxy value. (And also the http_proxy, In case of...)
• .build/venv/bin/pip uninstall transifex-client
• .build/venv/bin/pip install git+https://github.com/sbrunner/transifex-client.git@authenticated-proxy
• make -f svavoi.mk .build/locale/de/LC_MESSAGES/gmf.po )

[svavoi]

Hi Benjamin

Yes, I did already export the proxy like export https_proxy=https://<user>:<passwort>@<proxy>:<port>. I tried again to reinstall the transifex-client. But it is still not working...

svavoi@bvdgdi-svdmbap3:~/mapbs$ .build/venv/bin/pip install git+https://github.com/sbrunner/transifex-client.git@authenticated-proxy
Collecting git+https://github.com/sbrunner/transifex-client.git@authenticated-proxy
  Cloning https://github.com/sbrunner/transifex-client.git (to authenticated-proxy) to /tmp/pip-Mo570V-build
Requirement already satisfied: urllib3 in ./.build/venv/lib/python2.7/site-packages (from transifex-client==0.12.3)
Requirement already satisfied: six in ./.build/venv/lib/python2.7/site-packages (from transifex-client==0.12.3)
Installing collected packages: transifex-client
  Running setup.py install for transifex-client ... done
Successfully installed transifex-client-0.12.3
svavoi@bvdgdi-svdmbap3:~/mapbs$ make -f svavoi.mk .build/locale/de/LC_MESSAGES/gmf.po
mkdir -p .build/locale/de/LC_MESSAGES/
.build/venv/bin/tx pull -l de --force
MaxRetryError: HTTPSConnectionPool(host='www.transifex.com', port=443): Max retries exceeded with url: /api/2/project/ngeo/resource/ngeo-2_1/stats/ (Caused by ProxyError('Cannot connect to proxy.', error('Tunnel connection failed: 407 Proxy Authentication Required',)))
CONST_Makefile:751: recipe for target '.build/locale/de/LC_MESSAGES/gmf.po' failed
make: *** [.build/locale/de/LC_MESSAGES/gmf.po] Error 1

I've downloaded the file now by hand to continue with the update.

[svavoi]

I have now have the problem with the unique constraint on treeitem 'name':

sqlalchemy.exc.IntegrityError: (psycopg2.IntegrityError) could not create unique index "type_name_unique_treeitem"
DETAIL:  Key (type, name)=(group, Abwasserableitungsanlagen) is duplicated.
 [SQL: 'ALTER TABLE main.treeitem ADD CONSTRAINT type_name_unique_treeitem UNIQUE (type, name)']

Is it still true that I have to rename all the group layers which are not unique? (like mentioned here: https://github.com/camptocamp/c2cgeoportal/issues/2534)

[sbrunner]

Yes, but now WMS and WMTS layer had a layer(s) column to be able to have tow GeoMapFish layer with the same WMS layer.